petrel_client is not installed. If you read data locally instead of from ceph, ignore it. petrel_client is not installed. Using PIL to load images. 05/14/2026 20:19:48 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False 05/14/2026 20:19:48 - INFO - __main__ - Training/evaluation parameters TrainingArguments( _n_gpu=1, adafactor=False, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, auto_find_batch_size=False, bf16=True, bf16_full_eval=False, data_seed=None, dataloader_drop_last=False, dataloader_num_workers=4, dataloader_persistent_workers=False, dataloader_pin_memory=True, ddp_backend=None, ddp_broadcast_buffers=None, ddp_bucket_cap_mb=None, ddp_find_unused_parameters=None, ddp_timeout=1800, debug=[], deepspeed=./zero_stage1_config.json, disable_tqdm=False, dispatch_batches=None, do_eval=True, do_predict=False, do_train=True, eval_accumulation_steps=None, eval_delay=0, eval_steps=15, evaluation_strategy=IntervalStrategy.STEPS, fp16=False, fp16_backend=auto, fp16_full_eval=False, fp16_opt_level=O1, fsdp=[], fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False}, fsdp_min_num_params=0, fsdp_transformer_layer_cls_to_wrap=None, full_determinism=False, gradient_accumulation_steps=8, gradient_checkpointing=False, gradient_checkpointing_kwargs=None, greater_is_better=False, group_by_length=True, half_precision_backend=auto, hub_always_push=False, hub_model_id=None, hub_private_repo=False, hub_strategy=HubStrategy.EVERY_SAVE, hub_token=, ignore_data_skip=False, include_inputs_for_metrics=False, include_num_input_tokens_seen=False, include_tokens_per_second=False, jit_mode_eval=False, label_names=None, label_smoothing_factor=0.0, learning_rate=4e-05, length_column_name=length, load_best_model_at_end=True, local_rank=0, log_level=passive, log_level_replica=warning, log_on_each_node=True, logging_dir=work_dirs/internvl3_1b_walk_lora/runs/May14_20-19-47_91cd413e0006, logging_first_step=False, logging_nan_inf_filter=True, logging_steps=1.0, logging_strategy=IntervalStrategy.STEPS, lr_scheduler_kwargs={}, lr_scheduler_type=SchedulerType.COSINE, max_grad_norm=1.0, max_steps=-1, metric_for_best_model=eval_loss, mp_parameters=, neftune_noise_alpha=None, no_cuda=False, num_train_epochs=3.0, optim=OptimizerNames.ADAMW_TORCH, optim_args=None, output_dir=work_dirs/internvl3_1b_walk_lora, overwrite_output_dir=True, past_index=-1, per_device_eval_batch_size=2, per_device_train_batch_size=2, prediction_loss_only=False, push_to_hub=False, push_to_hub_model_id=None, push_to_hub_organization=None, push_to_hub_token=, ray_scope=last, remove_unused_columns=True, report_to=['wandb'], resume_from_checkpoint=None, run_name=work_dirs/internvl3_1b_walk_lora, save_on_each_node=False, save_only_model=False, save_safetensors=True, save_steps=15, save_strategy=IntervalStrategy.STEPS, save_total_limit=5, seed=42, skip_memory_metrics=True, split_batches=False, tf32=None, torch_compile=False, torch_compile_backend=None, torch_compile_mode=None, torchdynamo=None, tpu_metrics_debug=False, tpu_num_cores=None, use_cpu=False, use_ipex=False, use_legacy_prediction_loop=False, use_mps_device=False, warmup_ratio=0.03, warmup_steps=0, weight_decay=0.05, ) 05/14/2026 20:19:48 - INFO - __main__ - Loading Tokenizer: OpenGVLab/InternVL3-1B [INFO|tokenization_utils_base.py:2027] 2026-05-14 20:19:51,678 >> loading file vocab.json from cache at /root/.cache/huggingface/hub/models--OpenGVLab--InternVL3-1B/snapshots/4415a3b810e636d11dfa86b0e9ba40bb00535aa8/vocab.json [INFO|tokenization_utils_base.py:2027] 2026-05-14 20:19:51,678 >> loading file merges.txt from cache at /root/.cache/huggingface/hub/models--OpenGVLab--InternVL3-1B/snapshots/4415a3b810e636d11dfa86b0e9ba40bb00535aa8/merges.txt [INFO|tokenization_utils_base.py:2027] 2026-05-14 20:19:51,678 >> loading file added_tokens.json from cache at /root/.cache/huggingface/hub/models--OpenGVLab--InternVL3-1B/snapshots/4415a3b810e636d11dfa86b0e9ba40bb00535aa8/added_tokens.json [INFO|tokenization_utils_base.py:2027] 2026-05-14 20:19:51,678 >> loading file special_tokens_map.json from cache at /root/.cache/huggingface/hub/models--OpenGVLab--InternVL3-1B/snapshots/4415a3b810e636d11dfa86b0e9ba40bb00535aa8/special_tokens_map.json [INFO|tokenization_utils_base.py:2027] 2026-05-14 20:19:51,678 >> loading file tokenizer_config.json from cache at /root/.cache/huggingface/hub/models--OpenGVLab--InternVL3-1B/snapshots/4415a3b810e636d11dfa86b0e9ba40bb00535aa8/tokenizer_config.json [INFO|tokenization_utils_base.py:2027] 2026-05-14 20:19:51,678 >> loading file tokenizer.json from cache at /root/.cache/huggingface/hub/models--OpenGVLab--InternVL3-1B/snapshots/4415a3b810e636d11dfa86b0e9ba40bb00535aa8/tokenizer.json [WARNING|logging.py:314] 2026-05-14 20:19:51,954 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. 05/14/2026 20:19:51 - INFO - __main__ - Loading InternVLChatModel... [INFO|configuration_utils.py:729] 2026-05-14 20:19:52,377 >> loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--OpenGVLab--InternVL3-1B/snapshots/4415a3b810e636d11dfa86b0e9ba40bb00535aa8/config.json [INFO|configuration_utils.py:792] 2026-05-14 20:19:52,380 >> Model config InternVLChatConfig { "_commit_hash": "4415a3b810e636d11dfa86b0e9ba40bb00535aa8", "_name_or_path": "/mnt/petrelfs/wangweiyun/workspace_wwy/open_source/InternVL/internvl_chat/work_dirs/internvl_chat_v3_0/InternVL3_0-1B-MPO-try0-2", "architectures": [ "InternVLChatModel" ], "auto_map": { "AutoConfig": "OpenGVLab/InternVL3-1B--configuration_internvl_chat.InternVLChatConfig", "AutoModel": "OpenGVLab/InternVL3-1B--modeling_internvl_chat.InternVLChatModel", "AutoModelForCausalLM": "OpenGVLab/InternVL3-1B--modeling_internvl_chat.InternVLChatModel" }, "downsample_ratio": 0.5, "dynamic_image_size": true, "force_image_size": 448, "hidden_size": 896, "image_fold": null, "llm_config": { "_attn_implementation_autoset": true, "_name_or_path": "./pretrained/Qwen2.5-32B-Instruct", "add_cross_attention": false, "architectures": [ "Qwen2ForCausalLM" ], "attention_dropout": 0.0, "bad_words_ids": null, "begin_suppress_tokens": null, "bos_token_id": 151643, "chunk_size_feed_forward": 0, "cross_attention_hidden_size": null, "decoder_start_token_id": null, "diversity_penalty": 0.0, "do_sample": false, "early_stopping": false, "encoder_no_repeat_ngram_size": 0, "eos_token_id": 151643, "exponential_decay_length_penalty": null, "finetuning_task": null, "forced_bos_token_id": null, "forced_eos_token_id": null, "hidden_act": "silu", "hidden_size": 896, "id2label": { "0": "LABEL_0", "1": "LABEL_1" }, "initializer_range": 0.02, "intermediate_size": 4864, "is_decoder": false, "is_encoder_decoder": false, "label2id": { "LABEL_0": 0, "LABEL_1": 1 }, "length_penalty": 1.0, "max_length": 20, "max_position_embeddings": 32768, "max_window_layers": 70, "min_length": 0, "model_type": "qwen2", "moe_config": null, "no_repeat_ngram_size": 0, "num_attention_heads": 14, "num_beam_groups": 1, "num_beams": 1, "num_hidden_layers": 24, "num_key_value_heads": 2, "num_return_sequences": 1, "output_attentions": false, "output_hidden_states": false, "output_scores": false, "pad_token_id": null, "prefix": null, "problem_type": null, "pruned_heads": {}, "remove_invalid_values": false, "repetition_penalty": 1.0, "return_dict": true, "return_dict_in_generate": false, "rms_norm_eps": 1e-06, "rope_scaling": { "factor": 2.0, "rope_type": "dynamic", "type": "dynamic" }, "rope_theta": 1000000.0, "sep_token_id": null, "sliding_window": null, "suppress_tokens": null, "task_specific_params": null, "temperature": 1.0, "tf_legacy_loss": false, "tie_encoder_decoder": false, "tie_word_embeddings": false, "tokenizer_class": null, "top_k": 50, "top_p": 1.0, "torch_dtype": "bfloat16", "torchscript": false, "transformers_version": "4.37.2", "typical_p": 1.0, "use_bfloat16": true, "use_cache": false, "use_sliding_window": false, "vocab_size": 151674 }, "max_dynamic_patch": 12, "min_dynamic_patch": 1, "model_type": "internvl_chat", "pad2square": false, "ps_version": "v2", "select_layer": -1, "system_message": null, "template": "internvl2_5", "tie_word_embeddings": false, "torch_dtype": "bfloat16", "transformers_version": null, "use_backbone_lora": 0, "use_llm_lora": 0, "use_thumbnail": true, "vision_config": { "_attn_implementation_autoset": true, "_name_or_path": "OpenGVLab/InternViT-6B-448px-V1-5", "add_cross_attention": false, "architectures": [ "InternVisionModel" ], "attention_dropout": 0.0, "auto_map": { "AutoConfig": "configuration_intern_vit.InternVisionConfig", "AutoModel": "modeling_intern_vit.InternVisionModel" }, "bad_words_ids": null, "begin_suppress_tokens": null, "bos_token_id": null, "capacity_factor": 1.2, "chunk_size_feed_forward": 0, "cross_attention_hidden_size": null, "decoder_start_token_id": null, "diversity_penalty": 0.0, "do_sample": false, "drop_path_rate": 0.1, "dropout": 0.0, "early_stopping": false, "encoder_no_repeat_ngram_size": 0, "eos_token_id": null, "eval_capacity_factor": 1.4, "exponential_decay_length_penalty": null, "finetuning_task": null, "forced_bos_token_id": null, "forced_eos_token_id": null, "hidden_act": "gelu", "hidden_size": 1024, "id2label": { "0": "LABEL_0", "1": "LABEL_1" }, "image_size": 448, "initializer_factor": 0.1, "initializer_range": 1e-10, "intermediate_size": 4096, "is_decoder": false, "is_encoder_decoder": false, "label2id": { "LABEL_0": 0, "LABEL_1": 1 }, "laux_allreduce": "all_nodes", "layer_norm_eps": 1e-06, "length_penalty": 1.0, "max_length": 20, "min_length": 0, "model_type": "intern_vit_6b", "moe_coeff_ratio": 0.5, "moe_intermediate_size": 768, "moe_output_scale": 4.0, "no_repeat_ngram_size": 0, "noisy_gate_policy": "RSample_before", "norm_type": "layer_norm", "num_attention_heads": 16, "num_beam_groups": 1, "num_beams": 1, "num_channels": 3, "num_experts": 8, "num_hidden_layers": 24, "num_return_sequences": 1, "num_routed_experts": 4, "num_shared_experts": 4, "output_attentions": false, "output_hidden_states": false, "output_scores": false, "pad_token_id": null, "patch_size": 14, "prefix": null, "problem_type": null, "pruned_heads": {}, "qk_normalization": false, "qkv_bias": true, "remove_invalid_values": false, "repetition_penalty": 1.0, "return_dict": true, "return_dict_in_generate": false, "sep_token_id": null, "shared_expert_intermediate_size": 3072, "suppress_tokens": null, "task_specific_params": null, "temperature": 1.0, "tf_legacy_loss": false, "tie_encoder_decoder": false, "tie_word_embeddings": true, "tokenizer_class": null, "top_k": 50, "top_p": 1.0, "torch_dtype": "bfloat16", "torchscript": false, "transformers_version": "4.37.2", "typical_p": 1.0, "use_bfloat16": true, "use_flash_attn": true, "use_moe": false, "use_residual": true, "use_rts": false, "use_weighted_residual": false } } 05/14/2026 20:19:52 - INFO - __main__ - Using flash_attention_2 for LLaMA [INFO|modeling_utils.py:3476] 2026-05-14 20:19:57,795 >> loading weights file model.safetensors from cache at /root/.cache/huggingface/hub/models--OpenGVLab--InternVL3-1B/snapshots/4415a3b810e636d11dfa86b0e9ba40bb00535aa8/model.safetensors [INFO|modeling_utils.py:1426] 2026-05-14 20:19:57,817 >> Instantiating InternVLChatModel model under default dtype torch.bfloat16. [INFO|configuration_utils.py:826] 2026-05-14 20:19:57,818 >> Generate config GenerationConfig {} [INFO|configuration_utils.py:826] 2026-05-14 20:19:57,897 >> Generate config GenerationConfig { "bos_token_id": 151643, "eos_token_id": 151643, "use_cache": false } [INFO|modeling_utils.py:4350] 2026-05-14 20:20:00,284 >> All model checkpoint weights were used when initializing InternVLChatModel. [INFO|modeling_utils.py:4358] 2026-05-14 20:20:00,284 >> All the weights of InternVLChatModel were initialized from the model checkpoint at OpenGVLab/InternVL3-1B. If your task is similar to the task the model of the checkpoint was trained on, you can already use InternVLChatModel for predictions without further training. [INFO|configuration_utils.py:781] 2026-05-14 20:20:00,697 >> loading configuration file generation_config.json from cache at /root/.cache/huggingface/hub/models--OpenGVLab--InternVL3-1B/snapshots/4415a3b810e636d11dfa86b0e9ba40bb00535aa8/generation_config.json [INFO|configuration_utils.py:826] 2026-05-14 20:20:00,698 >> Generate config GenerationConfig {} 05/14/2026 20:20:00 - INFO - __main__ - Finished 05/14/2026 20:20:00 - INFO - __main__ - model.config.force_image_size: 448 05/14/2026 20:20:00 - INFO - __main__ - data_args.force_image_size: 448 05/14/2026 20:20:00 - INFO - __main__ - model.config.vision_config.image_size: 448 05/14/2026 20:20:00 - INFO - __main__ - max_dynamic_patch is set to 6 according to the meta file 05/14/2026 20:20:00 - INFO - __main__ - [Dataset] num_image_token: 256 05/14/2026 20:20:00 - INFO - __main__ - [Dataset] dynamic_image_size: True 05/14/2026 20:20:00 - INFO - __main__ - [Dataset] use_thumbnail: True 05/14/2026 20:20:00 - INFO - __main__ - [Dataset] min_dynamic_patch: 1, max_dynamic_patch: 6 05/14/2026 20:20:00 - INFO - __main__ - Formatting inputs...Skip in lazy mode 05/14/2026 20:20:00 - INFO - __main__ - Add dataset: walk-vlm-train with length: 15294 05/14/2026 20:20:00 - INFO - __main__ - Building evaluation dataset from: ./data/walk_vlm/walk_val_meta.json 05/14/2026 20:20:00 - INFO - __main__ - [Eval] max_dynamic_patch is set to 6 according to the meta file 05/14/2026 20:20:00 - INFO - __main__ - [Dataset] num_image_token: 256 05/14/2026 20:20:00 - INFO - __main__ - [Dataset] dynamic_image_size: True 05/14/2026 20:20:00 - INFO - __main__ - [Dataset] use_thumbnail: True 05/14/2026 20:20:00 - INFO - __main__ - [Dataset] min_dynamic_patch: 1, max_dynamic_patch: 6 05/14/2026 20:20:00 - INFO - __main__ - Formatting inputs...Skip in lazy mode 05/14/2026 20:20:00 - INFO - __main__ - [Eval] Add dataset: walk-vlm-val with length: 1700 05/14/2026 20:20:00 - INFO - __main__ - Evaluation dataset size: 1700 trainable params: 70,385,664 || all params: 700,083,584 || trainable%: 10.053894364704886 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.q_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.k_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.k_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.v_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.v_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.o_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.o_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.gate_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.gate_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.up_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.up_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.down_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.down_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.q_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.q_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.k_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.k_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.v_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.v_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.o_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.o_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.gate_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.gate_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.up_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.up_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.down_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.down_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.q_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.q_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.k_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.k_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.v_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.v_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.o_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.o_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.gate_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.gate_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.up_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.up_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.down_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.down_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.q_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.q_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.k_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.k_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.v_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.v_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.o_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.o_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.gate_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.gate_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.up_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.up_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.down_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.down_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.q_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.q_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.k_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.k_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.v_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.v_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.o_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.o_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.gate_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.gate_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.up_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.up_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.down_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.down_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.q_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.q_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.k_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.k_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.v_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.v_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.o_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.o_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.gate_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.gate_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.up_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.up_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.down_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.down_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.q_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.q_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.k_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.k_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.v_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.v_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.o_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.o_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.gate_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.gate_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.up_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.up_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.down_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.down_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.q_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.q_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.k_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.k_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.v_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.v_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.o_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.o_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.gate_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.gate_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.up_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.up_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.down_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.down_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.q_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.q_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.k_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.k_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.v_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.v_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.o_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.o_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.gate_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.gate_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.up_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.up_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.down_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.down_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.q_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.q_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.k_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.k_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.v_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.v_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.o_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.o_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.gate_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.gate_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.up_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.up_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.down_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.down_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.q_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.q_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.k_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.k_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.v_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.v_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.o_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.o_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.gate_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.gate_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.up_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.up_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.down_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.down_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.q_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.q_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.k_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.k_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.v_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.v_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.o_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.o_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.gate_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.gate_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.up_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.up_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.down_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.down_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.q_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.q_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.k_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.k_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.v_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.v_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.o_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.o_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.gate_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.gate_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.up_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.up_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.down_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.down_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.q_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.q_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.k_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.k_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.v_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.v_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.o_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.o_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.gate_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.gate_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.up_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.up_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.down_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.down_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.q_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.q_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.k_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.k_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.v_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.v_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.o_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.o_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.gate_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.gate_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.up_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.up_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.down_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.down_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.q_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.q_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.k_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.k_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.v_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.v_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.o_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.o_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.gate_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.gate_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.up_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.up_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.down_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.down_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.q_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.q_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.k_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.k_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.v_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.v_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.o_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.o_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.gate_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.gate_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.up_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.up_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.down_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.down_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.q_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.q_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.k_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.k_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.v_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.v_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.o_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.o_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.gate_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.gate_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.up_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.up_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.down_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.down_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.q_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.q_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.k_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.k_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.v_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.v_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.o_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.o_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.gate_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.gate_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.up_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.up_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.down_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.down_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.q_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.q_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.k_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.k_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.v_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.v_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.o_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.o_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.gate_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.gate_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.up_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.up_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.down_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.down_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.q_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.q_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.k_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.k_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.v_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.v_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.o_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.o_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.gate_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.gate_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.up_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.up_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.down_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.down_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.q_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.q_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.k_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.k_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.v_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.v_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.o_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.o_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.gate_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.gate_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.up_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.up_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.down_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.down_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.q_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.q_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.k_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.k_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.v_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.v_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.o_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.o_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.gate_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.gate_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.up_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.up_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.down_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.down_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.q_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.q_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.k_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.k_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.v_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.v_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.o_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.o_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.gate_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.gate_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.up_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.up_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.down_proj.lora_A.default.weight 05/14/2026 20:20:06 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.down_proj.lora_B.default.weight 05/14/2026 20:20:06 - INFO - __main__ - mlp1.0.weight 05/14/2026 20:20:06 - INFO - __main__ - mlp1.0.bias 05/14/2026 20:20:06 - INFO - __main__ - mlp1.1.weight 05/14/2026 20:20:06 - INFO - __main__ - mlp1.1.bias 05/14/2026 20:20:06 - INFO - __main__ - mlp1.3.weight 05/14/2026 20:20:06 - INFO - __main__ - mlp1.3.bias 05/14/2026 20:20:06 - INFO - __main__ - ✅ Early stopping enabled: patience=3, threshold=0.005 [INFO|trainer.py:571] 2026-05-14 20:20:06,519 >> Using auto half precision backend [INFO|trainer.py:1721] 2026-05-14 20:20:40,504 >> ***** Running training ***** [INFO|trainer.py:1722] 2026-05-14 20:20:40,505 >> Num examples = 15,294 [INFO|trainer.py:1723] 2026-05-14 20:20:40,505 >> Num Epochs = 3 [INFO|trainer.py:1724] 2026-05-14 20:20:40,505 >> Instantaneous batch size per device = 2 [INFO|trainer.py:1727] 2026-05-14 20:20:40,505 >> Total train batch size (w. parallel, distributed & accumulation) = 16 [INFO|trainer.py:1728] 2026-05-14 20:20:40,505 >> Gradient Accumulation steps = 8 [INFO|trainer.py:1729] 2026-05-14 20:20:40,505 >> Total optimization steps = 2,865 [INFO|trainer.py:1730] 2026-05-14 20:20:40,514 >> Number of trainable parameters = 74,868,480 [INFO|integration_utils.py:722] 2026-05-14 20:20:40,519 >> Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true" wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /root/.netrc. wandb: Currently logged in as: rpgkr3 (vlm-blind-assist) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin wandb: Tracking run with wandb version 0.27.0 wandb: Run data is saved locally in /workspace/InternVL/internvl_chat/wandb/run-20260514_202041-ampnlkg8 wandb: Run `wandb offline` to turn off syncing. wandb: Syncing run InternVL3-1B_walkvlm_lora_earlystop wandb: ⭐️ View project at https://wandb.ai/vlm-blind-assist/internvl-walk wandb: 🚀 View run at https://wandb.ai/vlm-blind-assist/internvl-walk/runs/ampnlkg8 0%| | 0/2865 [00:00> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 20:21:40,412 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 20:21:40,413 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-15 [INFO|configuration_utils.py:473] 2026-05-14 20:26:48,712 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-15/config.json [INFO|configuration_utils.py:594] 2026-05-14 20:26:48,723 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-15/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 20:26:54,614 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-15/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 20:26:54,625 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-15/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 20:26:54,632 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-15/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 20:26:54,634 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-15/added_tokens.json dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 304598.38 | bwd_microstep: 221.23 | bwd_inner_microstep: 220.79 | bwd_allreduce_microstep: 0.21 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 174.00 | bwd_microstep: 256.21 | bwd_inner_microstep: 255.95 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 174.24 | bwd_microstep: 233.41 | bwd_inner_microstep: 233.04 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 169.68 | bwd_microstep: 235.28 | bwd_inner_microstep: 235.16 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 163.98 | bwd_microstep: 233.74 | bwd_inner_microstep: 233.54 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 164.31 | bwd_microstep: 232.27 | bwd_inner_microstep: 232.07 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 163.70 | bwd_microstep: 235.11 | bwd_inner_microstep: 234.64 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.96 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.52 | bwd_microstep: 277.85 | bwd_inner_microstep: 267.50 | bwd_allreduce_microstep: 10.27 | step_microstep: 21.68 [Rank 0] time (ms) | fwd: 305758.30 | bwd: 1925.32 | bwd_inner: 1913.12 | bwd_allreduce: 11.16 | step: 22.36 1%| | 16/2865 [06:24<79:41:59, 100.71s/it] {'loss': 2.358, 'learning_rate': 7.44186046511628e-06, 'epoch': 0.02} 1%| | 16/2865 [06:24<79:41:59, 100.71s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 163.05 | bwd_microstep: 224.38 | bwd_inner_microstep: 224.25 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 164.52 | bwd_microstep: 232.10 | bwd_inner_microstep: 231.74 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 169.69 | bwd_microstep: 223.94 | bwd_inner_microstep: 223.54 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 164.56 | bwd_microstep: 233.48 | bwd_inner_microstep: 233.32 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 173.51 | bwd_microstep: 232.48 | bwd_inner_microstep: 232.18 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 168.03 | bwd_microstep: 223.90 | bwd_inner_microstep: 223.47 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 163.03 | bwd_microstep: 235.71 | bwd_inner_microstep: 235.36 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 163.09 | bwd_microstep: 267.73 | bwd_inner_microstep: 256.53 | bwd_allreduce_microstep: 11.01 | step_microstep: 21.36 [Rank 0] time (ms) | fwd: 1329.34 | bwd: 1873.90 | bwd_inner: 1860.97 | bwd_allreduce: 11.78 | step: 22.16 1%| | 17/2865 [06:27<56:31:12, 71.44s/it] {'loss': 2.1133, 'learning_rate': 7.906976744186048e-06, 'epoch': 0.02} 1%| | 17/2865 [06:27<56:31:12, 71.44s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 167.41 | bwd_microstep: 223.55 | bwd_inner_microstep: 223.18 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 168.91 | bwd_microstep: 243.54 | bwd_inner_microstep: 243.38 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 168.31 | bwd_microstep: 252.31 | bwd_inner_microstep: 252.02 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 174.09 | bwd_microstep: 234.33 | bwd_inner_microstep: 234.22 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 168.83 | bwd_microstep: 246.49 | bwd_inner_microstep: 246.20 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 171.44 | bwd_microstep: 245.35 | bwd_inner_microstep: 245.00 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 163.73 | bwd_microstep: 231.55 | bwd_inner_microstep: 231.45 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.36 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 176.66 | bwd_microstep: 269.94 | bwd_inner_microstep: 255.13 | bwd_allreduce_microstep: 14.59 | step_microstep: 28.67 [Rank 0] time (ms) | fwd: 1359.24 | bwd: 1947.18 | bwd_inner: 1930.97 | bwd_allreduce: 15.35 | step: 29.28 1%| | 18/2865 [06:30<40:21:26, 51.03s/it] {'loss': 2.3785, 'learning_rate': 8.372093023255815e-06, 'epoch': 0.02} 1%| | 18/2865 [06:30<40:21:26, 51.03s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 166.20 | bwd_microstep: 236.13 | bwd_inner_microstep: 235.94 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 168.19 | bwd_microstep: 229.58 | bwd_inner_microstep: 229.18 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 172.55 | bwd_microstep: 242.14 | bwd_inner_microstep: 241.82 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 163.77 | bwd_microstep: 236.95 | bwd_inner_microstep: 236.59 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 163.58 | bwd_microstep: 256.81 | bwd_inner_microstep: 256.47 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 165.76 | bwd_microstep: 261.90 | bwd_inner_microstep: 261.49 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 165.25 | bwd_microstep: 239.54 | bwd_inner_microstep: 239.15 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.30 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 169.63 | bwd_microstep: 269.25 | bwd_inner_microstep: 258.78 | bwd_allreduce_microstep: 10.38 | step_microstep: 22.13 [Rank 0] time (ms) | fwd: 1334.81 | bwd: 1972.50 | bwd_inner: 1959.94 | bwd_allreduce: 11.37 | step: 23.00 1%| | 19/2865 [06:34<29:03:31, 36.76s/it] {'loss': 2.051, 'learning_rate': 8.837209302325582e-06, 'epoch': 0.02} 1%| | 19/2865 [06:34<29:03:31, 36.76s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 163.37 | bwd_microstep: 223.23 | bwd_inner_microstep: 223.06 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 163.32 | bwd_microstep: 242.58 | bwd_inner_microstep: 242.34 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 166.01 | bwd_microstep: 264.18 | bwd_inner_microstep: 264.06 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 167.13 | bwd_microstep: 294.30 | bwd_inner_microstep: 294.07 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 181.32 | bwd_microstep: 282.46 | bwd_inner_microstep: 282.28 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 177.53 | bwd_microstep: 291.12 | bwd_inner_microstep: 290.95 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 180.22 | bwd_microstep: 285.96 | bwd_inner_microstep: 285.59 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.07 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 180.20 | bwd_microstep: 339.71 | bwd_inner_microstep: 324.03 | bwd_allreduce_microstep: 15.52 | step_microstep: 24.74 [Rank 0] time (ms) | fwd: 1378.95 | bwd: 2223.67 | bwd_inner: 2206.72 | bwd_allreduce: 16.14 | step: 25.41 1%| | 20/2865 [06:38<21:13:54, 26.87s/it] {'loss': 1.8168, 'learning_rate': 9.30232558139535e-06, 'epoch': 0.02} 1%| | 20/2865 [06:38<21:13:54, 26.87s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 192.36 | bwd_microstep: 296.20 | bwd_inner_microstep: 295.97 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 185.92 | bwd_microstep: 301.97 | bwd_inner_microstep: 301.76 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 180.83 | bwd_microstep: 286.09 | bwd_inner_microstep: 285.55 | bwd_allreduce_microstep: 0.22 | step_microstep: 0.28 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 185.35 | bwd_microstep: 285.26 | bwd_inner_microstep: 284.96 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 177.26 | bwd_microstep: 287.98 | bwd_inner_microstep: 287.76 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 191.21 | bwd_microstep: 291.90 | bwd_inner_microstep: 291.62 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 178.01 | bwd_microstep: 250.87 | bwd_inner_microstep: 250.69 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.28 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 179.37 | bwd_microstep: 275.74 | bwd_inner_microstep: 265.00 | bwd_allreduce_microstep: 10.61 | step_microstep: 22.57 [Rank 0] time (ms) | fwd: 1470.17 | bwd: 2276.15 | bwd_inner: 2263.81 | bwd_allreduce: 11.36 | step: 23.41 1%| | 21/2865 [06:42<15:47:36, 19.99s/it] {'loss': 2.0361, 'learning_rate': 9.767441860465117e-06, 'epoch': 0.02} 1%| | 21/2865 [06:42<15:47:36, 19.99s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 170.22 | bwd_microstep: 234.92 | bwd_inner_microstep: 234.68 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 168.74 | bwd_microstep: 300.37 | bwd_inner_microstep: 300.22 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 201.26 | bwd_microstep: 286.25 | bwd_inner_microstep: 285.83 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 179.06 | bwd_microstep: 297.39 | bwd_inner_microstep: 297.08 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 190.22 | bwd_microstep: 309.27 | bwd_inner_microstep: 309.00 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 190.55 | bwd_microstep: 272.29 | bwd_inner_microstep: 271.72 | bwd_allreduce_microstep: 0.23 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 167.60 | bwd_microstep: 268.79 | bwd_inner_microstep: 268.26 | bwd_allreduce_microstep: 0.23 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.10 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 167.43 | bwd_microstep: 281.72 | bwd_inner_microstep: 262.10 | bwd_allreduce_microstep: 19.49 | step_microstep: 22.77 [Rank 0] time (ms) | fwd: 1434.89 | bwd: 2251.19 | bwd_inner: 2229.44 | bwd_allreduce: 20.48 | step: 23.87 1%| | 22/2865 [06:46<11:58:49, 15.17s/it] {'loss': 2.0988, 'learning_rate': 1.0232558139534884e-05, 'epoch': 0.02} 1%| | 22/2865 [06:46<11:58:49, 15.17s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 185.98 | bwd_microstep: 283.80 | bwd_inner_microstep: 283.44 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 203.93 | bwd_microstep: 272.46 | bwd_inner_microstep: 271.94 | bwd_allreduce_microstep: 0.22 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 167.21 | bwd_microstep: 263.17 | bwd_inner_microstep: 262.94 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 177.59 | bwd_microstep: 286.38 | bwd_inner_microstep: 285.96 | bwd_allreduce_microstep: 0.20 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 198.08 | bwd_microstep: 268.76 | bwd_inner_microstep: 268.37 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 186.52 | bwd_microstep: 282.71 | bwd_inner_microstep: 282.55 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 187.92 | bwd_microstep: 307.87 | bwd_inner_microstep: 307.48 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.11 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 187.61 | bwd_microstep: 318.31 | bwd_inner_microstep: 306.34 | bwd_allreduce_microstep: 11.79 | step_microstep: 23.08 [Rank 0] time (ms) | fwd: 1494.69 | bwd: 2283.71 | bwd_inner: 2269.55 | bwd_allreduce: 12.92 | step: 24.28 1%| | 23/2865 [06:50<9:19:49, 11.82s/it] {'loss': 1.8503, 'learning_rate': 1.0697674418604651e-05, 'epoch': 0.02} 1%| | 23/2865 [06:50<9:19:49, 11.82s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 176.36 | bwd_microstep: 275.12 | bwd_inner_microstep: 274.75 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 176.74 | bwd_microstep: 290.40 | bwd_inner_microstep: 290.23 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 177.84 | bwd_microstep: 271.98 | bwd_inner_microstep: 271.72 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 179.27 | bwd_microstep: 299.70 | bwd_inner_microstep: 299.53 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 187.81 | bwd_microstep: 328.79 | bwd_inner_microstep: 328.51 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 188.90 | bwd_microstep: 287.68 | bwd_inner_microstep: 287.41 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 195.19 | bwd_microstep: 272.88 | bwd_inner_microstep: 272.68 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.39 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 176.01 | bwd_microstep: 349.01 | bwd_inner_microstep: 334.06 | bwd_allreduce_microstep: 14.81 | step_microstep: 23.28 [Rank 0] time (ms) | fwd: 1457.89 | bwd: 2375.72 | bwd_inner: 2359.25 | bwd_allreduce: 15.54 | step: 24.12 1%| | 24/2865 [06:54<7:29:32, 9.49s/it] {'loss': 2.0808, 'learning_rate': 1.116279069767442e-05, 'epoch': 0.03} 1%| | 24/2865 [06:54<7:29:32, 9.49s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 946 [Rank 0] time (ms) | fwd_microstep: 202.08 | bwd_microstep: 285.20 | bwd_inner_microstep: 284.98 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 204.46 | bwd_microstep: 291.45 | bwd_inner_microstep: 291.13 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 184.18 | bwd_microstep: 256.28 | bwd_inner_microstep: 256.06 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 188.77 | bwd_microstep: 280.49 | bwd_inner_microstep: 280.07 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 175.90 | bwd_microstep: 238.03 | bwd_inner_microstep: 237.65 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 169.94 | bwd_microstep: 276.69 | bwd_inner_microstep: 276.47 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 182.84 | bwd_microstep: 263.68 | bwd_inner_microstep: 263.49 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.95 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 163.00 | bwd_microstep: 287.42 | bwd_inner_microstep: 276.24 | bwd_allreduce_microstep: 11.08 | step_microstep: 22.20 [Rank 0] time (ms) | fwd: 1471.01 | bwd: 2179.42 | bwd_inner: 2166.50 | bwd_allreduce: 12.01 | step: 23.03 1%| | 25/2865 [06:58<6:09:18, 7.80s/it] {'loss': 1.9016, 'learning_rate': 1.1627906976744187e-05, 'epoch': 0.03} 1%| | 25/2865 [06:58<6:09:18, 7.80s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 958 [Rank 0] time (ms) | fwd_microstep: 183.18 | bwd_microstep: 260.00 | bwd_inner_microstep: 259.85 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 192.77 | bwd_microstep: 266.69 | bwd_inner_microstep: 266.26 | bwd_allreduce_microstep: 0.21 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 175.24 | bwd_microstep: 275.73 | bwd_inner_microstep: 275.42 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 190.16 | bwd_microstep: 289.53 | bwd_inner_microstep: 289.14 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 183.68 | bwd_microstep: 274.78 | bwd_inner_microstep: 274.34 | bwd_allreduce_microstep: 0.21 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 177.71 | bwd_microstep: 243.68 | bwd_inner_microstep: 243.48 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 909 [Rank 0] time (ms) | fwd_microstep: 178.79 | bwd_microstep: 280.59 | bwd_inner_microstep: 280.38 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.37 | bwd_microstep: 270.81 | bwd_inner_microstep: 258.78 | bwd_allreduce_microstep: 11.86 | step_microstep: 21.30 [Rank 0] time (ms) | fwd: 1444.69 | bwd: 2161.99 | bwd_inner: 2148.10 | bwd_allreduce: 12.78 | step: 22.24 1%| | 26/2865 [07:01<5:12:52, 6.61s/it] {'loss': 1.7763, 'learning_rate': 1.2093023255813954e-05, 'epoch': 0.03} 1%| | 26/2865 [07:01<5:12:52, 6.61s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 190.78 | bwd_microstep: 267.83 | bwd_inner_microstep: 267.51 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 180.59 | bwd_microstep: 284.08 | bwd_inner_microstep: 283.74 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 179.54 | bwd_microstep: 332.39 | bwd_inner_microstep: 332.07 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 181.63 | bwd_microstep: 297.51 | bwd_inner_microstep: 297.32 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 173.75 | bwd_microstep: 283.91 | bwd_inner_microstep: 283.74 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 180.06 | bwd_microstep: 291.63 | bwd_inner_microstep: 291.31 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 192.67 | bwd_microstep: 273.90 | bwd_inner_microstep: 273.68 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.27 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 177.81 | bwd_microstep: 348.82 | bwd_inner_microstep: 329.65 | bwd_allreduce_microstep: 19.05 | step_microstep: 25.40 [Rank 0] time (ms) | fwd: 1456.66 | bwd: 2380.21 | bwd_inner: 2359.45 | bwd_allreduce: 19.88 | step: 26.20 1%| | 27/2865 [07:05<4:36:50, 5.85s/it] {'loss': 1.9512, 'learning_rate': 1.2558139534883723e-05, 'epoch': 0.03} 1%| | 27/2865 [07:05<4:36:50, 5.85s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 186.72 | bwd_microstep: 268.88 | bwd_inner_microstep: 268.55 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 196.10 | bwd_microstep: 285.48 | bwd_inner_microstep: 285.33 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 186.06 | bwd_microstep: 297.55 | bwd_inner_microstep: 297.38 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 197.22 | bwd_microstep: 270.92 | bwd_inner_microstep: 270.73 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 176.76 | bwd_microstep: 242.06 | bwd_inner_microstep: 241.67 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 167.02 | bwd_microstep: 224.23 | bwd_inner_microstep: 224.10 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 167.62 | bwd_microstep: 218.22 | bwd_inner_microstep: 218.11 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.30 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 169.47 | bwd_microstep: 262.04 | bwd_inner_microstep: 248.65 | bwd_allreduce_microstep: 13.27 | step_microstep: 23.16 [Rank 0] time (ms) | fwd: 1446.79 | bwd: 2069.52 | bwd_inner: 2054.80 | bwd_allreduce: 13.96 | step: 23.95 1%| | 28/2865 [07:09<4:06:45, 5.22s/it] {'loss': 1.8621, 'learning_rate': 1.302325581395349e-05, 'epoch': 0.03} 1%| | 28/2865 [07:09<4:06:45, 5.22s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 164.01 | bwd_microstep: 235.36 | bwd_inner_microstep: 235.26 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 162.42 | bwd_microstep: 233.76 | bwd_inner_microstep: 233.58 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 161.59 | bwd_microstep: 222.05 | bwd_inner_microstep: 221.91 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 175.30 | bwd_microstep: 217.55 | bwd_inner_microstep: 217.19 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 162.42 | bwd_microstep: 237.80 | bwd_inner_microstep: 237.46 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.62 | bwd_microstep: 237.26 | bwd_inner_microstep: 237.10 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.01 | bwd_microstep: 245.06 | bwd_inner_microstep: 244.87 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 146.56 | bwd_microstep: 233.21 | bwd_inner_microstep: 222.32 | bwd_allreduce_microstep: 10.79 | step_microstep: 21.40 [Rank 0] time (ms) | fwd: 1293.79 | bwd: 1862.19 | bwd_inner: 1849.94 | bwd_allreduce: 11.46 | step: 22.16 1%| | 29/2865 [07:13<3:39:55, 4.65s/it] {'loss': 1.5368, 'learning_rate': 1.3488372093023257e-05, 'epoch': 0.03} 1%| | 29/2865 [07:13<3:39:55, 4.65s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 165.58 | bwd_microstep: 220.34 | bwd_inner_microstep: 219.94 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 162.51 | bwd_microstep: 226.39 | bwd_inner_microstep: 226.20 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 163.09 | bwd_microstep: 227.58 | bwd_inner_microstep: 227.17 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 162.32 | bwd_microstep: 231.07 | bwd_inner_microstep: 230.78 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 167.64 | bwd_microstep: 230.39 | bwd_inner_microstep: 229.90 | bwd_allreduce_microstep: 0.21 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.73 | bwd_microstep: 224.80 | bwd_inner_microstep: 224.55 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 172.80 | bwd_microstep: 240.59 | bwd_inner_microstep: 240.38 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.13 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 171.92 | bwd_microstep: 266.64 | bwd_inner_microstep: 254.94 | bwd_allreduce_microstep: 11.61 | step_microstep: 21.92 [Rank 0] time (ms) | fwd: 1326.42 | bwd: 1868.00 | bwd_inner: 1854.36 | bwd_allreduce: 12.52 | step: 22.82 1%| | 30/2865 [07:16<3:22:16, 4.28s/it] {'loss': 1.9195, 'learning_rate': 1.3953488372093025e-05, 'epoch': 0.03} 1%| | 30/2865 [07:16<3:22:16, 4.28s/it][INFO|trainer.py:3242] 2026-05-14 20:27:59,144 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 20:27:59,144 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 20:27:59,144 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-30 [INFO|configuration_utils.py:473] 2026-05-14 20:33:07,541 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-30/config.json [INFO|configuration_utils.py:594] 2026-05-14 20:33:07,552 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-30/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 20:33:12,478 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-30/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 20:33:12,489 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-30/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 20:33:12,497 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-30/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 20:33:12,500 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-30/added_tokens.json dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 305025.41 | bwd_microstep: 311.31 | bwd_inner_microstep: 311.04 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 181.60 | bwd_microstep: 305.87 | bwd_inner_microstep: 305.50 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 192.09 | bwd_microstep: 265.15 | bwd_inner_microstep: 265.00 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 177.73 | bwd_microstep: 238.53 | bwd_inner_microstep: 238.35 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 187.46 | bwd_microstep: 245.16 | bwd_inner_microstep: 245.01 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 179.69 | bwd_microstep: 255.13 | bwd_inner_microstep: 254.99 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 167.19 | bwd_microstep: 277.21 | bwd_inner_microstep: 276.92 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.10 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 177.94 | bwd_microstep: 372.48 | bwd_inner_microstep: 354.19 | bwd_allreduce_microstep: 18.18 | step_microstep: 22.31 [Rank 0] time (ms) | fwd: 306276.92 | bwd: 2270.98 | bwd_inner: 2251.36 | bwd_allreduce: 18.88 | step: 23.09 1%| | 31/2865 [12:41<79:00:58, 100.37s/it] {'loss': 1.5574, 'learning_rate': 1.441860465116279e-05, 'epoch': 0.03} 1%| | 31/2865 [12:41<79:00:58, 100.37s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 177.62 | bwd_microstep: 345.12 | bwd_inner_microstep: 344.90 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 199.85 | bwd_microstep: 264.39 | bwd_inner_microstep: 264.18 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 165.18 | bwd_microstep: 244.79 | bwd_inner_microstep: 244.56 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 177.21 | bwd_microstep: 272.93 | bwd_inner_microstep: 272.63 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 177.67 | bwd_microstep: 272.16 | bwd_inner_microstep: 272.00 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 177.13 | bwd_microstep: 295.70 | bwd_inner_microstep: 295.41 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 157.93 | bwd_microstep: 236.76 | bwd_inner_microstep: 236.61 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.98 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 166.81 | bwd_microstep: 298.72 | bwd_inner_microstep: 285.99 | bwd_allreduce_microstep: 12.52 | step_microstep: 21.55 [Rank 0] time (ms) | fwd: 1399.27 | bwd: 2230.69 | bwd_inner: 2216.75 | bwd_allreduce: 13.13 | step: 22.28 1%| | 32/2865 [12:44<56:11:48, 71.41s/it] {'loss': 1.5251, 'learning_rate': 1.488372093023256e-05, 'epoch': 0.03} 1%| | 32/2865 [12:44<56:11:48, 71.41s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 182.25 | bwd_microstep: 276.68 | bwd_inner_microstep: 276.44 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 186.27 | bwd_microstep: 270.31 | bwd_inner_microstep: 270.15 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 185.15 | bwd_microstep: 279.88 | bwd_inner_microstep: 279.59 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 167.84 | bwd_microstep: 261.40 | bwd_inner_microstep: 261.24 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 197.14 | bwd_microstep: 309.64 | bwd_inner_microstep: 309.44 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 207.29 | bwd_microstep: 305.62 | bwd_inner_microstep: 305.43 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 191.41 | bwd_microstep: 317.57 | bwd_inner_microstep: 317.30 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 179.58 | bwd_microstep: 361.26 | bwd_inner_microstep: 349.92 | bwd_allreduce_microstep: 11.24 | step_microstep: 22.48 [Rank 0] time (ms) | fwd: 1496.75 | bwd: 2382.46 | bwd_inner: 2369.85 | bwd_allreduce: 11.89 | step: 23.20 1%| | 33/2865 [12:48<40:17:17, 51.21s/it] {'loss': 1.5617, 'learning_rate': 1.5348837209302328e-05, 'epoch': 0.03} 1%| | 33/2865 [12:48<40:17:17, 51.21s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 175.69 | bwd_microstep: 322.55 | bwd_inner_microstep: 322.27 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 167.80 | bwd_microstep: 245.52 | bwd_inner_microstep: 245.32 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 181.90 | bwd_microstep: 287.31 | bwd_inner_microstep: 287.16 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 186.77 | bwd_microstep: 292.70 | bwd_inner_microstep: 292.28 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.24 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 205.88 | bwd_microstep: 293.22 | bwd_inner_microstep: 292.84 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 180.34 | bwd_microstep: 290.36 | bwd_inner_microstep: 290.20 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 189.88 | bwd_microstep: 259.91 | bwd_inner_microstep: 259.75 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.02 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 171.09 | bwd_microstep: 346.24 | bwd_inner_microstep: 334.77 | bwd_allreduce_microstep: 11.36 | step_microstep: 22.06 [Rank 0] time (ms) | fwd: 1459.21 | bwd: 2337.97 | bwd_inner: 2324.94 | bwd_allreduce: 12.13 | step: 22.99 1%| | 34/2865 [12:52<29:08:05, 37.05s/it] {'loss': 1.4944, 'learning_rate': 1.5813953488372095e-05, 'epoch': 0.04} 1%| | 34/2865 [12:52<29:08:05, 37.05s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 184.12 | bwd_microstep: 305.74 | bwd_inner_microstep: 305.41 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 180.85 | bwd_microstep: 292.47 | bwd_inner_microstep: 292.15 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 178.92 | bwd_microstep: 249.83 | bwd_inner_microstep: 249.60 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 179.31 | bwd_microstep: 279.94 | bwd_inner_microstep: 279.76 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 185.92 | bwd_microstep: 279.09 | bwd_inner_microstep: 278.90 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 180.80 | bwd_microstep: 280.33 | bwd_inner_microstep: 279.94 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 176.59 | bwd_microstep: 277.06 | bwd_inner_microstep: 276.66 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.36 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 189.81 | bwd_microstep: 288.96 | bwd_inner_microstep: 278.48 | bwd_allreduce_microstep: 10.32 | step_microstep: 22.81 [Rank 0] time (ms) | fwd: 1456.17 | bwd: 2253.58 | bwd_inner: 2241.36 | bwd_allreduce: 11.14 | step: 23.85 1%| | 35/2865 [12:56<21:18:57, 27.12s/it] {'loss': 1.442, 'learning_rate': 1.6279069767441862e-05, 'epoch': 0.04} 1%| | 35/2865 [12:56<21:18:57, 27.12s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 940 [Rank 0] time (ms) | fwd_microstep: 168.45 | bwd_microstep: 257.33 | bwd_inner_microstep: 257.18 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 179.66 | bwd_microstep: 299.04 | bwd_inner_microstep: 298.85 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 177.48 | bwd_microstep: 281.60 | bwd_inner_microstep: 281.44 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 192.14 | bwd_microstep: 301.60 | bwd_inner_microstep: 301.45 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 179.31 | bwd_microstep: 281.23 | bwd_inner_microstep: 281.05 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 191.33 | bwd_microstep: 306.49 | bwd_inner_microstep: 306.24 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 164.02 | bwd_microstep: 248.12 | bwd_inner_microstep: 247.99 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.02 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 172.84 | bwd_microstep: 335.15 | bwd_inner_microstep: 320.45 | bwd_allreduce_microstep: 14.46 | step_microstep: 22.17 [Rank 0] time (ms) | fwd: 1425.10 | bwd: 2310.66 | bwd_inner: 2294.94 | bwd_allreduce: 15.02 | step: 22.82 1%|▏ | 36/2865 [13:00<15:50:26, 20.16s/it] {'loss': 1.542, 'learning_rate': 1.674418604651163e-05, 'epoch': 0.04} 1%|▏ | 36/2865 [13:00<15:50:26, 20.16s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 952 [Rank 0] time (ms) | fwd_microstep: 174.83 | bwd_microstep: 268.41 | bwd_inner_microstep: 268.20 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 177.63 | bwd_microstep: 270.14 | bwd_inner_microstep: 269.94 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 176.04 | bwd_microstep: 265.96 | bwd_inner_microstep: 265.82 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 162.99 | bwd_microstep: 225.70 | bwd_inner_microstep: 225.51 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 161.95 | bwd_microstep: 225.99 | bwd_inner_microstep: 225.83 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 160.59 | bwd_microstep: 242.65 | bwd_inner_microstep: 242.45 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 163.82 | bwd_microstep: 241.48 | bwd_inner_microstep: 241.23 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.82 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 149.52 | bwd_microstep: 228.29 | bwd_inner_microstep: 218.05 | bwd_allreduce_microstep: 10.10 | step_microstep: 21.25 [Rank 0] time (ms) | fwd: 1327.24 | bwd: 1968.73 | bwd_inner: 1957.40 | bwd_allreduce: 10.58 | step: 21.95 1%|▏ | 37/2865 [13:04<11:54:07, 15.15s/it] {'loss': 1.4577, 'learning_rate': 1.7209302325581396e-05, 'epoch': 0.04} 1%|▏ | 37/2865 [13:04<11:54:07, 15.15s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 164.73 | bwd_microstep: 223.14 | bwd_inner_microstep: 222.98 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 164.30 | bwd_microstep: 225.89 | bwd_inner_microstep: 225.76 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 166.71 | bwd_microstep: 241.49 | bwd_inner_microstep: 241.31 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 164.11 | bwd_microstep: 224.65 | bwd_inner_microstep: 224.49 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 164.00 | bwd_microstep: 223.24 | bwd_inner_microstep: 223.05 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 161.72 | bwd_microstep: 225.50 | bwd_inner_microstep: 225.12 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 162.08 | bwd_microstep: 226.58 | bwd_inner_microstep: 226.38 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 159.86 | bwd_microstep: 261.94 | bwd_inner_microstep: 251.78 | bwd_allreduce_microstep: 10.08 | step_microstep: 21.18 [Rank 0] time (ms) | fwd: 1307.38 | bwd: 1852.57 | bwd_inner: 1841.13 | bwd_allreduce: 10.68 | step: 21.86 1%|▏ | 38/2865 [13:07<9:07:45, 11.63s/it] {'loss': 1.477, 'learning_rate': 1.7674418604651163e-05, 'epoch': 0.04} 1%|▏ | 38/2865 [13:07<9:07:45, 11.63s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 175.30 | bwd_microstep: 226.48 | bwd_inner_microstep: 226.25 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.72 | bwd_microstep: 226.37 | bwd_inner_microstep: 226.00 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.82 | bwd_microstep: 233.01 | bwd_inner_microstep: 232.62 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 163.05 | bwd_microstep: 225.65 | bwd_inner_microstep: 225.34 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 179.21 | bwd_microstep: 226.89 | bwd_inner_microstep: 226.58 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 162.94 | bwd_microstep: 231.20 | bwd_inner_microstep: 231.00 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 164.44 | bwd_microstep: 224.40 | bwd_inner_microstep: 224.32 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 166.33 | bwd_microstep: 284.90 | bwd_inner_microstep: 275.22 | bwd_allreduce_microstep: 9.60 | step_microstep: 21.18 [Rank 0] time (ms) | fwd: 1333.66 | bwd: 1879.05 | bwd_inner: 1867.69 | bwd_allreduce: 10.48 | step: 21.92 1%|▏ | 39/2865 [13:11<7:11:14, 9.16s/it] {'loss': 1.641, 'learning_rate': 1.813953488372093e-05, 'epoch': 0.04} 1%|▏ | 39/2865 [13:11<7:11:14, 9.16s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 163.50 | bwd_microstep: 251.34 | bwd_inner_microstep: 251.25 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.81 | bwd_microstep: 235.57 | bwd_inner_microstep: 235.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 167.35 | bwd_microstep: 259.19 | bwd_inner_microstep: 259.10 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 168.98 | bwd_microstep: 226.57 | bwd_inner_microstep: 226.44 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 164.10 | bwd_microstep: 225.83 | bwd_inner_microstep: 225.71 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 165.53 | bwd_microstep: 223.97 | bwd_inner_microstep: 223.84 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 164.43 | bwd_microstep: 225.76 | bwd_inner_microstep: 225.56 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 162.21 | bwd_microstep: 263.00 | bwd_inner_microstep: 252.15 | bwd_allreduce_microstep: 10.76 | step_microstep: 21.11 [Rank 0] time (ms) | fwd: 1316.80 | bwd: 1911.32 | bwd_inner: 1899.65 | bwd_allreduce: 11.20 | step: 21.65 1%|▏ | 40/2865 [13:14<5:49:41, 7.43s/it] {'loss': 1.4285, 'learning_rate': 1.86046511627907e-05, 'epoch': 0.04} 1%|▏ | 40/2865 [13:14<5:49:41, 7.43s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 171.66 | bwd_microstep: 239.74 | bwd_inner_microstep: 239.49 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 188.11 | bwd_microstep: 224.83 | bwd_inner_microstep: 224.66 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 164.22 | bwd_microstep: 229.10 | bwd_inner_microstep: 228.73 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 163.27 | bwd_microstep: 236.87 | bwd_inner_microstep: 236.51 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 148.30 | bwd_microstep: 199.12 | bwd_inner_microstep: 198.86 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 164.38 | bwd_microstep: 233.31 | bwd_inner_microstep: 233.03 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 162.71 | bwd_microstep: 224.95 | bwd_inner_microstep: 224.55 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 167.60 | bwd_microstep: 257.01 | bwd_inner_microstep: 246.66 | bwd_allreduce_microstep: 10.22 | step_microstep: 21.02 [Rank 0] time (ms) | fwd: 1330.13 | bwd: 1845.09 | bwd_inner: 1832.89 | bwd_allreduce: 11.15 | step: 21.84 1%|▏ | 41/2865 [13:17<4:52:08, 6.21s/it] {'loss': 1.2625, 'learning_rate': 1.9069767441860468e-05, 'epoch': 0.04} 1%|▏ | 41/2865 [13:17<4:52:08, 6.21s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 940 [Rank 0] time (ms) | fwd_microstep: 164.56 | bwd_microstep: 229.65 | bwd_inner_microstep: 229.47 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 164.96 | bwd_microstep: 231.73 | bwd_inner_microstep: 231.61 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 164.46 | bwd_microstep: 227.96 | bwd_inner_microstep: 227.76 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 177.76 | bwd_microstep: 223.38 | bwd_inner_microstep: 223.18 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.57 | bwd_microstep: 222.94 | bwd_inner_microstep: 222.81 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.41 | bwd_microstep: 238.68 | bwd_inner_microstep: 238.39 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 146.76 | bwd_microstep: 224.35 | bwd_inner_microstep: 224.11 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 147.80 | bwd_microstep: 245.51 | bwd_inner_microstep: 235.15 | bwd_allreduce_microstep: 10.28 | step_microstep: 21.32 [Rank 0] time (ms) | fwd: 1293.16 | bwd: 1844.31 | bwd_inner: 1832.81 | bwd_allreduce: 10.83 | step: 21.88 1%|▏ | 42/2865 [13:21<4:11:01, 5.34s/it] {'loss': 1.2509, 'learning_rate': 1.9534883720930235e-05, 'epoch': 0.04} 1%|▏ | 42/2865 [13:21<4:11:01, 5.34s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 166.10 | bwd_microstep: 219.62 | bwd_inner_microstep: 219.40 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 163.36 | bwd_microstep: 225.89 | bwd_inner_microstep: 225.63 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 164.86 | bwd_microstep: 224.23 | bwd_inner_microstep: 224.14 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 158.81 | bwd_microstep: 197.82 | bwd_inner_microstep: 197.56 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 164.36 | bwd_microstep: 223.53 | bwd_inner_microstep: 223.23 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 166.73 | bwd_microstep: 231.76 | bwd_inner_microstep: 231.63 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 164.95 | bwd_microstep: 225.64 | bwd_inner_microstep: 225.29 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 166.68 | bwd_microstep: 274.51 | bwd_inner_microstep: 264.50 | bwd_allreduce_microstep: 9.93 | step_microstep: 21.38 [Rank 0] time (ms) | fwd: 1315.73 | bwd: 1823.17 | bwd_inner: 1811.67 | bwd_allreduce: 10.64 | step: 22.21 2%|▏ | 43/2865 [13:24<3:42:30, 4.73s/it] {'loss': 1.2406, 'learning_rate': 2e-05, 'epoch': 0.04} 2%|▏ | 43/2865 [13:24<3:42:30, 4.73s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 164.56 | bwd_microstep: 236.03 | bwd_inner_microstep: 235.68 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 162.81 | bwd_microstep: 230.75 | bwd_inner_microstep: 230.64 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 163.52 | bwd_microstep: 225.48 | bwd_inner_microstep: 225.38 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 165.69 | bwd_microstep: 229.28 | bwd_inner_microstep: 228.90 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 163.76 | bwd_microstep: 254.71 | bwd_inner_microstep: 254.41 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 167.79 | bwd_microstep: 258.93 | bwd_inner_microstep: 258.68 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 167.32 | bwd_microstep: 246.00 | bwd_inner_microstep: 245.85 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.98 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 167.92 | bwd_microstep: 299.44 | bwd_inner_microstep: 283.71 | bwd_allreduce_microstep: 15.58 | step_microstep: 22.39 [Rank 0] time (ms) | fwd: 1323.22 | bwd: 1980.74 | bwd_inner: 1963.61 | bwd_allreduce: 16.32 | step: 23.11 2%|▏ | 44/2865 [13:27<3:24:48, 4.36s/it] {'loss': 1.3302, 'learning_rate': 2.046511627906977e-05, 'epoch': 0.05} 2%|▏ | 44/2865 [13:27<3:24:48, 4.36s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 947 [Rank 0] time (ms) | fwd_microstep: 180.17 | bwd_microstep: 249.23 | bwd_inner_microstep: 248.98 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 169.09 | bwd_microstep: 237.15 | bwd_inner_microstep: 236.97 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 162.82 | bwd_microstep: 234.38 | bwd_inner_microstep: 234.22 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 166.65 | bwd_microstep: 234.09 | bwd_inner_microstep: 233.76 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 168.64 | bwd_microstep: 231.90 | bwd_inner_microstep: 231.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 164.19 | bwd_microstep: 231.60 | bwd_inner_microstep: 231.33 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 164.16 | bwd_microstep: 234.38 | bwd_inner_microstep: 234.13 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.06 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 168.12 | bwd_microstep: 286.41 | bwd_inner_microstep: 273.84 | bwd_allreduce_microstep: 12.47 | step_microstep: 24.22 [Rank 0] time (ms) | fwd: 1343.71 | bwd: 1939.26 | bwd_inner: 1925.36 | bwd_allreduce: 13.15 | step: 24.79 2%|▏ | 45/2865 [13:31<3:12:14, 4.09s/it] {'loss': 1.499, 'learning_rate': 2.093023255813954e-05, 'epoch': 0.05} 2%|▏ | 45/2865 [13:31<3:12:14, 4.09s/it][INFO|trainer.py:3242] 2026-05-14 20:34:14,098 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 20:34:14,098 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 20:34:14,099 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-45 [INFO|configuration_utils.py:473] 2026-05-14 20:39:19,385 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-45/config.json [INFO|configuration_utils.py:594] 2026-05-14 20:39:19,391 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-45/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 20:39:24,464 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-45/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 20:39:24,472 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-45/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 20:39:24,478 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-45/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 20:39:24,479 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-45/added_tokens.json dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 302431.97 | bwd_microstep: 262.62 | bwd_inner_microstep: 262.44 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.23 | bwd_microstep: 257.62 | bwd_inner_microstep: 257.48 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 938 [Rank 0] time (ms) | fwd_microstep: 163.91 | bwd_microstep: 240.75 | bwd_inner_microstep: 240.35 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.80 | bwd_microstep: 219.53 | bwd_inner_microstep: 219.16 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 163.08 | bwd_microstep: 213.31 | bwd_inner_microstep: 213.08 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.14 | bwd_microstep: 216.62 | bwd_inner_microstep: 216.32 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 160.38 | bwd_microstep: 214.80 | bwd_inner_microstep: 214.47 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.48 | bwd_microstep: 266.20 | bwd_inner_microstep: 255.84 | bwd_allreduce_microstep: 10.21 | step_microstep: 21.33 [Rank 0] time (ms) | fwd: 303551.14 | bwd: 1891.62 | bwd_inner: 1879.59 | bwd_allreduce: 11.02 | step: 22.10 2%|▏ | 46/2865 [18:53<78:00:51, 99.63s/it] {'loss': 1.3516, 'learning_rate': 2.1395348837209303e-05, 'epoch': 0.05} 2%|▏ | 46/2865 [18:53<78:00:51, 99.63s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 162.47 | bwd_microstep: 215.57 | bwd_inner_microstep: 215.27 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 161.23 | bwd_microstep: 217.45 | bwd_inner_microstep: 217.22 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 161.54 | bwd_microstep: 219.65 | bwd_inner_microstep: 219.27 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.39 | bwd_microstep: 218.12 | bwd_inner_microstep: 217.76 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.72 | bwd_microstep: 214.14 | bwd_inner_microstep: 213.77 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.64 | bwd_microstep: 218.83 | bwd_inner_microstep: 218.65 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 159.95 | bwd_microstep: 216.23 | bwd_inner_microstep: 216.08 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 159.74 | bwd_microstep: 248.17 | bwd_inner_microstep: 238.18 | bwd_allreduce_microstep: 9.86 | step_microstep: 21.17 [Rank 0] time (ms) | fwd: 1287.57 | bwd: 1768.31 | bwd_inner: 1756.63 | bwd_allreduce: 10.68 | step: 21.95 2%|▏ | 47/2865 [18:57<55:20:54, 70.71s/it] {'loss': 1.3749, 'learning_rate': 2.186046511627907e-05, 'epoch': 0.05} 2%|▏ | 47/2865 [18:57<55:20:54, 70.71s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 160.80 | bwd_microstep: 212.01 | bwd_inner_microstep: 211.65 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 160.68 | bwd_microstep: 219.18 | bwd_inner_microstep: 219.05 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 160.38 | bwd_microstep: 213.80 | bwd_inner_microstep: 213.68 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 160.76 | bwd_microstep: 215.95 | bwd_inner_microstep: 215.65 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 161.15 | bwd_microstep: 220.65 | bwd_inner_microstep: 220.27 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.26 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 164.04 | bwd_microstep: 213.68 | bwd_inner_microstep: 213.56 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 162.00 | bwd_microstep: 214.43 | bwd_inner_microstep: 214.04 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 161.99 | bwd_microstep: 246.58 | bwd_inner_microstep: 236.70 | bwd_allreduce_microstep: 9.76 | step_microstep: 21.11 [Rank 0] time (ms) | fwd: 1291.68 | bwd: 1756.47 | bwd_inner: 1745.00 | bwd_allreduce: 10.54 | step: 22.06 2%|▏ | 48/2865 [19:00<39:29:14, 50.46s/it] {'loss': 1.507, 'learning_rate': 2.232558139534884e-05, 'epoch': 0.05} 2%|▏ | 48/2865 [19:00<39:29:14, 50.46s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 161.16 | bwd_microstep: 215.29 | bwd_inner_microstep: 214.91 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 940 [Rank 0] time (ms) | fwd_microstep: 161.00 | bwd_microstep: 218.82 | bwd_inner_microstep: 218.49 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 162.90 | bwd_microstep: 231.34 | bwd_inner_microstep: 230.94 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 160.47 | bwd_microstep: 214.97 | bwd_inner_microstep: 214.64 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.27 | bwd_microstep: 216.45 | bwd_inner_microstep: 216.22 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.08 | bwd_microstep: 216.83 | bwd_inner_microstep: 216.71 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 147.28 | bwd_microstep: 173.53 | bwd_inner_microstep: 173.40 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 160.03 | bwd_microstep: 216.78 | bwd_inner_microstep: 207.19 | bwd_allreduce_microstep: 9.51 | step_microstep: 21.28 [Rank 0] time (ms) | fwd: 1274.07 | bwd: 1704.16 | bwd_inner: 1692.91 | bwd_allreduce: 10.34 | step: 22.08 2%|▏ | 49/2865 [19:03<28:22:10, 36.27s/it] {'loss': 1.5157, 'learning_rate': 2.2790697674418607e-05, 'epoch': 0.05} 2%|▏ | 49/2865 [19:03<28:22:10, 36.27s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 161.25 | bwd_microstep: 215.59 | bwd_inner_microstep: 215.35 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 160.90 | bwd_microstep: 217.36 | bwd_inner_microstep: 217.01 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.18 | bwd_microstep: 214.73 | bwd_inner_microstep: 214.45 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 160.43 | bwd_microstep: 218.12 | bwd_inner_microstep: 217.88 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 146.41 | bwd_microstep: 191.07 | bwd_inner_microstep: 190.93 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.74 | bwd_microstep: 217.98 | bwd_inner_microstep: 217.58 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 159.61 | bwd_microstep: 220.49 | bwd_inner_microstep: 220.10 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.22 | bwd_microstep: 245.76 | bwd_inner_microstep: 235.96 | bwd_allreduce_microstep: 9.65 | step_microstep: 21.24 [Rank 0] time (ms) | fwd: 1269.63 | bwd: 1741.25 | bwd_inner: 1729.76 | bwd_allreduce: 10.43 | step: 21.95 2%|▏ | 50/2865 [19:06<20:35:39, 26.34s/it] {'loss': 1.2619, 'learning_rate': 2.3255813953488374e-05, 'epoch': 0.05} 2%|▏ | 50/2865 [19:06<20:35:39, 26.34s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 160.86 | bwd_microstep: 230.83 | bwd_inner_microstep: 230.70 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 944 [Rank 0] time (ms) | fwd_microstep: 162.61 | bwd_microstep: 221.18 | bwd_inner_microstep: 220.83 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.62 | bwd_microstep: 221.98 | bwd_inner_microstep: 221.79 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 160.92 | bwd_microstep: 218.26 | bwd_inner_microstep: 218.13 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 161.85 | bwd_microstep: 216.35 | bwd_inner_microstep: 216.23 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.94 | bwd_microstep: 194.07 | bwd_inner_microstep: 193.94 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.15 | bwd_microstep: 187.12 | bwd_inner_microstep: 186.91 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.28 | bwd_microstep: 228.23 | bwd_inner_microstep: 215.25 | bwd_allreduce_microstep: 12.80 | step_microstep: 21.27 [Rank 0] time (ms) | fwd: 1290.14 | bwd: 1718.14 | bwd_inner: 1704.15 | bwd_allreduce: 13.26 | step: 21.88 2%|▏ | 51/2865 [19:09<15:09:17, 19.39s/it] {'loss': 1.2551, 'learning_rate': 2.3720930232558144e-05, 'epoch': 0.05} 2%|▏ | 51/2865 [19:09<15:09:17, 19.39s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 161.56 | bwd_microstep: 215.14 | bwd_inner_microstep: 214.94 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 159.86 | bwd_microstep: 189.58 | bwd_inner_microstep: 189.18 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 159.97 | bwd_microstep: 193.20 | bwd_inner_microstep: 193.09 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.43 | bwd_microstep: 192.20 | bwd_inner_microstep: 191.85 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.23 | bwd_microstep: 186.39 | bwd_inner_microstep: 186.03 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.07 | bwd_microstep: 212.43 | bwd_inner_microstep: 212.09 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.02 | bwd_microstep: 202.13 | bwd_inner_microstep: 201.81 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 159.77 | bwd_microstep: 235.25 | bwd_inner_microstep: 225.69 | bwd_allreduce_microstep: 9.48 | step_microstep: 21.01 [Rank 0] time (ms) | fwd: 1282.80 | bwd: 1626.48 | bwd_inner: 1615.11 | bwd_allreduce: 10.37 | step: 21.85 2%|▏ | 52/2865 [19:12<11:19:21, 14.49s/it] {'loss': 1.586, 'learning_rate': 2.4186046511627908e-05, 'epoch': 0.05} 2%|▏ | 52/2865 [19:12<11:19:21, 14.49s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 938 [Rank 0] time (ms) | fwd_microstep: 161.32 | bwd_microstep: 224.89 | bwd_inner_microstep: 224.51 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.28 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 162.01 | bwd_microstep: 230.62 | bwd_inner_microstep: 230.34 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 164.62 | bwd_microstep: 223.21 | bwd_inner_microstep: 222.70 | bwd_allreduce_microstep: 0.25 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 167.19 | bwd_microstep: 229.41 | bwd_inner_microstep: 229.16 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 148.57 | bwd_microstep: 202.86 | bwd_inner_microstep: 202.66 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 164.91 | bwd_microstep: 223.95 | bwd_inner_microstep: 223.54 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 162.80 | bwd_microstep: 222.63 | bwd_inner_microstep: 222.49 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.42 | bwd_microstep: 247.50 | bwd_inner_microstep: 237.75 | bwd_allreduce_microstep: 9.66 | step_microstep: 21.30 [Rank 0] time (ms) | fwd: 1291.73 | bwd: 1805.30 | bwd_inner: 1793.60 | bwd_allreduce: 10.55 | step: 22.19 2%|▏ | 53/2865 [19:16<8:41:23, 11.12s/it] {'loss': 1.4297, 'learning_rate': 2.4651162790697675e-05, 'epoch': 0.06} 2%|▏ | 53/2865 [19:16<8:41:23, 11.12s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 160.73 | bwd_microstep: 213.79 | bwd_inner_microstep: 213.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.11 | bwd_microstep: 215.53 | bwd_inner_microstep: 215.47 | bwd_allreduce_microstep: 0.03 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.37 | bwd_microstep: 191.19 | bwd_inner_microstep: 191.17 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 155.21 | bwd_microstep: 189.47 | bwd_inner_microstep: 189.45 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 155.37 | bwd_microstep: 190.50 | bwd_inner_microstep: 190.49 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 143.55 | bwd_microstep: 174.38 | bwd_inner_microstep: 174.36 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 154.95 | bwd_microstep: 192.74 | bwd_inner_microstep: 192.71 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.79 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 155.51 | bwd_microstep: 216.81 | bwd_inner_microstep: 208.50 | bwd_allreduce_microstep: 8.24 | step_microstep: 20.93 [Rank 0] time (ms) | fwd: 1245.70 | bwd: 1584.45 | bwd_inner: 1575.88 | bwd_allreduce: 8.36 | step: 21.40 2%|▏ | 54/2865 [19:19<6:46:37, 8.68s/it] {'loss': 1.1373, 'learning_rate': 2.5116279069767445e-05, 'epoch': 0.06} 2%|▏ | 54/2865 [19:19<6:46:37, 8.68s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 158.65 | bwd_microstep: 194.44 | bwd_inner_microstep: 194.39 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 156.46 | bwd_microstep: 192.28 | bwd_inner_microstep: 192.26 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 156.08 | bwd_microstep: 189.37 | bwd_inner_microstep: 189.35 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 156.22 | bwd_microstep: 187.42 | bwd_inner_microstep: 187.39 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 156.39 | bwd_microstep: 190.15 | bwd_inner_microstep: 190.12 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 155.81 | bwd_microstep: 187.93 | bwd_inner_microstep: 187.91 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 143.93 | bwd_microstep: 176.98 | bwd_inner_microstep: 176.93 | bwd_allreduce_microstep: 0.03 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.78 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 155.56 | bwd_microstep: 241.10 | bwd_inner_microstep: 232.91 | bwd_allreduce_microstep: 8.12 | step_microstep: 21.08 [Rank 0] time (ms) | fwd: 1239.02 | bwd: 1559.71 | bwd_inner: 1551.29 | bwd_allreduce: 8.20 | step: 21.54 2%|▏ | 55/2865 [19:22<5:25:48, 6.96s/it] {'loss': 1.3681, 'learning_rate': 2.5581395348837212e-05, 'epoch': 0.06} 2%|▏ | 55/2865 [19:22<5:25:48, 6.96s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 155.63 | bwd_microstep: 216.07 | bwd_inner_microstep: 216.00 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 155.72 | bwd_microstep: 249.09 | bwd_inner_microstep: 248.98 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 159.73 | bwd_microstep: 220.19 | bwd_inner_microstep: 220.09 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 166.26 | bwd_microstep: 222.94 | bwd_inner_microstep: 222.78 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 148.62 | bwd_microstep: 199.99 | bwd_inner_microstep: 199.78 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.79 | bwd_microstep: 192.14 | bwd_inner_microstep: 191.92 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 162.39 | bwd_microstep: 192.57 | bwd_inner_microstep: 192.22 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 163.66 | bwd_microstep: 256.24 | bwd_inner_microstep: 243.93 | bwd_allreduce_microstep: 12.14 | step_microstep: 21.21 [Rank 0] time (ms) | fwd: 1272.70 | bwd: 1749.32 | bwd_inner: 1735.98 | bwd_allreduce: 12.65 | step: 21.80 2%|▏ | 56/2865 [19:25<4:32:39, 5.82s/it] {'loss': 1.4136, 'learning_rate': 2.604651162790698e-05, 'epoch': 0.06} 2%|▏ | 56/2865 [19:25<4:32:39, 5.82s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 940 [Rank 0] time (ms) | fwd_microstep: 171.50 | bwd_microstep: 231.15 | bwd_inner_microstep: 230.77 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 169.67 | bwd_microstep: 232.29 | bwd_inner_microstep: 232.14 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 164.06 | bwd_microstep: 235.06 | bwd_inner_microstep: 234.89 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 163.59 | bwd_microstep: 233.70 | bwd_inner_microstep: 233.32 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 169.74 | bwd_microstep: 231.02 | bwd_inner_microstep: 230.68 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 161.58 | bwd_microstep: 241.43 | bwd_inner_microstep: 241.28 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.96 | bwd_microstep: 222.78 | bwd_inner_microstep: 222.65 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.97 | bwd_microstep: 287.99 | bwd_inner_microstep: 277.87 | bwd_allreduce_microstep: 10.05 | step_microstep: 21.04 [Rank 0] time (ms) | fwd: 1322.96 | bwd: 1915.52 | bwd_inner: 1903.92 | bwd_allreduce: 10.80 | step: 21.68 2%|▏ | 57/2865 [19:28<3:58:35, 5.10s/it] {'loss': 1.3521, 'learning_rate': 2.6511627906976743e-05, 'epoch': 0.06} 2%|▏ | 57/2865 [19:28<3:58:35, 5.10s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 162.21 | bwd_microstep: 225.51 | bwd_inner_microstep: 225.12 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 164.58 | bwd_microstep: 222.83 | bwd_inner_microstep: 222.62 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 163.00 | bwd_microstep: 224.28 | bwd_inner_microstep: 223.98 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 164.25 | bwd_microstep: 225.58 | bwd_inner_microstep: 225.46 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 162.91 | bwd_microstep: 223.20 | bwd_inner_microstep: 222.82 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.26 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 166.21 | bwd_microstep: 224.36 | bwd_inner_microstep: 224.06 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 162.50 | bwd_microstep: 224.73 | bwd_inner_microstep: 224.62 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 164.84 | bwd_microstep: 257.66 | bwd_inner_microstep: 247.30 | bwd_allreduce_microstep: 10.27 | step_microstep: 20.99 [Rank 0] time (ms) | fwd: 1310.39 | bwd: 1828.29 | bwd_inner: 1816.38 | bwd_allreduce: 11.01 | step: 21.87 2%|▏ | 58/2865 [19:32<3:33:15, 4.56s/it] {'loss': 1.0214, 'learning_rate': 2.6976744186046514e-05, 'epoch': 0.06} 2%|▏ | 58/2865 [19:32<3:33:15, 4.56s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 164.59 | bwd_microstep: 229.99 | bwd_inner_microstep: 229.78 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 167.69 | bwd_microstep: 245.36 | bwd_inner_microstep: 245.03 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 189.11 | bwd_microstep: 229.34 | bwd_inner_microstep: 229.13 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.68 | bwd_microstep: 220.28 | bwd_inner_microstep: 220.08 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.70 | bwd_microstep: 234.61 | bwd_inner_microstep: 234.20 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 167.61 | bwd_microstep: 229.57 | bwd_inner_microstep: 229.42 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 163.97 | bwd_microstep: 235.50 | bwd_inner_microstep: 235.30 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.19 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.51 | bwd_microstep: 276.37 | bwd_inner_microstep: 266.48 | bwd_allreduce_microstep: 9.81 | step_microstep: 21.61 [Rank 0] time (ms) | fwd: 1338.75 | bwd: 1901.16 | bwd_inner: 1889.75 | bwd_allreduce: 10.57 | step: 22.46 2%|▏ | 59/2865 [19:35<3:17:14, 4.22s/it] {'loss': 1.3137, 'learning_rate': 2.744186046511628e-05, 'epoch': 0.06} 2%|▏ | 59/2865 [19:35<3:17:14, 4.22s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 175.17 | bwd_microstep: 224.91 | bwd_inner_microstep: 224.80 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 164.16 | bwd_microstep: 220.25 | bwd_inner_microstep: 220.15 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 163.27 | bwd_microstep: 224.19 | bwd_inner_microstep: 223.94 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 165.72 | bwd_microstep: 226.15 | bwd_inner_microstep: 225.96 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 163.42 | bwd_microstep: 225.90 | bwd_inner_microstep: 225.65 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 173.12 | bwd_microstep: 227.17 | bwd_inner_microstep: 227.02 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 162.39 | bwd_microstep: 226.08 | bwd_inner_microstep: 225.98 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 161.73 | bwd_microstep: 296.98 | bwd_inner_microstep: 287.20 | bwd_allreduce_microstep: 9.71 | step_microstep: 21.49 [Rank 0] time (ms) | fwd: 1328.85 | bwd: 1871.73 | bwd_inner: 1860.88 | bwd_allreduce: 10.23 | step: 22.16 2%|▏ | 60/2865 [19:38<3:05:23, 3.97s/it] {'loss': 1.0428, 'learning_rate': 2.790697674418605e-05, 'epoch': 0.06} 2%|▏ | 60/2865 [19:38<3:05:23, 3.97s/it][INFO|trainer.py:3242] 2026-05-14 20:40:21,512 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 20:40:21,513 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 20:40:21,513 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-60 [INFO|configuration_utils.py:473] 2026-05-14 20:45:27,888 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-60/config.json [INFO|configuration_utils.py:594] 2026-05-14 20:45:27,899 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-60/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 20:45:33,009 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-60/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 20:45:33,026 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-60/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 20:45:33,035 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-60/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 20:45:33,037 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-60/added_tokens.json dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 943 [Rank 0] time (ms) | fwd_microstep: 302865.84 | bwd_microstep: 235.98 | bwd_inner_microstep: 235.84 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 164.30 | bwd_microstep: 222.12 | bwd_inner_microstep: 221.97 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 163.20 | bwd_microstep: 228.53 | bwd_inner_microstep: 228.39 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 164.11 | bwd_microstep: 239.37 | bwd_inner_microstep: 238.97 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 172.90 | bwd_microstep: 230.82 | bwd_inner_microstep: 230.64 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 172.37 | bwd_microstep: 229.64 | bwd_inner_microstep: 229.26 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 172.19 | bwd_microstep: 279.79 | bwd_inner_microstep: 279.51 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.11 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 185.23 | bwd_microstep: 314.85 | bwd_inner_microstep: 302.09 | bwd_allreduce_microstep: 12.60 | step_microstep: 22.29 [Rank 0] time (ms) | fwd: 304049.39 | bwd: 1981.24 | bwd_inner: 1966.98 | bwd_allreduce: 13.32 | step: 22.95 2%|▏ | 61/2865 [25:01<77:28:54, 99.48s/it] {'loss': 1.1287, 'learning_rate': 2.8372093023255818e-05, 'epoch': 0.06} 2%|▏ | 61/2865 [25:01<77:28:54, 99.48s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 938 [Rank 0] time (ms) | fwd_microstep: 178.05 | bwd_microstep: 240.60 | bwd_inner_microstep: 240.40 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 162.93 | bwd_microstep: 255.81 | bwd_inner_microstep: 255.54 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 182.23 | bwd_microstep: 256.44 | bwd_inner_microstep: 256.24 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 174.54 | bwd_microstep: 274.71 | bwd_inner_microstep: 274.47 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 208.88 | bwd_microstep: 287.59 | bwd_inner_microstep: 287.43 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 175.36 | bwd_microstep: 282.20 | bwd_inner_microstep: 281.95 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 155.42 | bwd_microstep: 216.41 | bwd_inner_microstep: 216.18 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 166.97 | bwd_microstep: 276.70 | bwd_inner_microstep: 266.87 | bwd_allreduce_microstep: 9.75 | step_microstep: 21.48 [Rank 0] time (ms) | fwd: 1404.25 | bwd: 2090.56 | bwd_inner: 2079.44 | bwd_allreduce: 10.39 | step: 22.21 2%|▏ | 62/2865 [25:04<55:04:59, 70.75s/it] {'loss': 1.1939, 'learning_rate': 2.883720930232558e-05, 'epoch': 0.06} 2%|▏ | 62/2865 [25:04<55:04:59, 70.75s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 170.69 | bwd_microstep: 231.94 | bwd_inner_microstep: 231.74 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 176.08 | bwd_microstep: 292.71 | bwd_inner_microstep: 292.48 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 177.05 | bwd_microstep: 270.33 | bwd_inner_microstep: 270.12 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 174.85 | bwd_microstep: 278.04 | bwd_inner_microstep: 277.87 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 176.01 | bwd_microstep: 272.25 | bwd_inner_microstep: 271.99 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 163.76 | bwd_microstep: 248.98 | bwd_inner_microstep: 248.85 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 169.49 | bwd_microstep: 269.38 | bwd_inner_microstep: 269.08 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.96 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 157.56 | bwd_microstep: 269.66 | bwd_inner_microstep: 258.82 | bwd_allreduce_microstep: 10.75 | step_microstep: 21.42 [Rank 0] time (ms) | fwd: 1365.38 | bwd: 2133.41 | bwd_inner: 2121.23 | bwd_allreduce: 11.42 | step: 22.26 2%|▏ | 63/2865 [25:08<39:24:47, 50.64s/it] {'loss': 1.4606, 'learning_rate': 2.930232558139535e-05, 'epoch': 0.07} 2%|▏ | 63/2865 [25:08<39:24:47, 50.64s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 941 [Rank 0] time (ms) | fwd_microstep: 207.02 | bwd_microstep: 259.68 | bwd_inner_microstep: 259.45 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 178.08 | bwd_microstep: 306.88 | bwd_inner_microstep: 306.70 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 185.24 | bwd_microstep: 284.53 | bwd_inner_microstep: 284.38 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 178.83 | bwd_microstep: 309.24 | bwd_inner_microstep: 309.02 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 169.67 | bwd_microstep: 249.87 | bwd_inner_microstep: 249.72 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 165.93 | bwd_microstep: 254.23 | bwd_inner_microstep: 253.84 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 180.55 | bwd_microstep: 273.22 | bwd_inner_microstep: 272.81 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.11 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 188.13 | bwd_microstep: 313.28 | bwd_inner_microstep: 302.23 | bwd_allreduce_microstep: 10.96 | step_microstep: 29.00 [Rank 0] time (ms) | fwd: 1453.34 | bwd: 2251.11 | bwd_inner: 2238.54 | bwd_allreduce: 11.69 | step: 29.81 2%|▏ | 64/2865 [25:12<28:29:38, 36.62s/it] {'loss': 1.1316, 'learning_rate': 2.976744186046512e-05, 'epoch': 0.07} 2%|▏ | 64/2865 [25:12<28:29:38, 36.62s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 174.87 | bwd_microstep: 242.02 | bwd_inner_microstep: 241.81 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 180.08 | bwd_microstep: 278.46 | bwd_inner_microstep: 278.21 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 165.70 | bwd_microstep: 254.04 | bwd_inner_microstep: 253.89 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 179.74 | bwd_microstep: 272.00 | bwd_inner_microstep: 271.84 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 182.51 | bwd_microstep: 311.73 | bwd_inner_microstep: 311.55 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 160.94 | bwd_microstep: 289.37 | bwd_inner_microstep: 289.20 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 179.59 | bwd_microstep: 340.57 | bwd_inner_microstep: 340.31 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.96 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 178.95 | bwd_microstep: 389.13 | bwd_inner_microstep: 378.15 | bwd_allreduce_microstep: 10.89 | step_microstep: 21.48 [Rank 0] time (ms) | fwd: 1402.26 | bwd: 2377.43 | bwd_inner: 2365.24 | bwd_allreduce: 11.48 | step: 22.17 2%|▏ | 65/2865 [25:16<20:51:54, 26.83s/it] {'loss': 1.2673, 'learning_rate': 3.0232558139534886e-05, 'epoch': 0.07} 2%|▏ | 65/2865 [25:16<20:51:54, 26.83s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 198.13 | bwd_microstep: 261.73 | bwd_inner_microstep: 261.52 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 182.50 | bwd_microstep: 267.05 | bwd_inner_microstep: 266.63 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 165.94 | bwd_microstep: 250.04 | bwd_inner_microstep: 249.78 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 179.48 | bwd_microstep: 276.69 | bwd_inner_microstep: 276.45 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 182.63 | bwd_microstep: 285.03 | bwd_inner_microstep: 284.71 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 173.98 | bwd_microstep: 235.29 | bwd_inner_microstep: 235.08 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 177.86 | bwd_microstep: 271.63 | bwd_inner_microstep: 271.32 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.23 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 203.57 | bwd_microstep: 323.25 | bwd_inner_microstep: 308.62 | bwd_allreduce_microstep: 14.52 | step_microstep: 24.61 [Rank 0] time (ms) | fwd: 1463.93 | bwd: 2170.86 | bwd_inner: 2154.58 | bwd_allreduce: 15.30 | step: 25.49 2%|▏ | 66/2865 [25:20<15:29:57, 19.93s/it] {'loss': 1.3902, 'learning_rate': 3.0697674418604656e-05, 'epoch': 0.07} 2%|▏ | 66/2865 [25:20<15:29:57, 19.93s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 180.40 | bwd_microstep: 280.90 | bwd_inner_microstep: 280.73 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 183.12 | bwd_microstep: 283.56 | bwd_inner_microstep: 283.30 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 191.93 | bwd_microstep: 273.37 | bwd_inner_microstep: 273.05 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 178.73 | bwd_microstep: 245.42 | bwd_inner_microstep: 245.13 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 176.07 | bwd_microstep: 289.79 | bwd_inner_microstep: 289.56 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 184.90 | bwd_microstep: 249.15 | bwd_inner_microstep: 248.97 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 168.42 | bwd_microstep: 249.69 | bwd_inner_microstep: 249.52 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.54 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 166.53 | bwd_microstep: 285.57 | bwd_inner_microstep: 272.70 | bwd_allreduce_microstep: 12.64 | step_microstep: 27.98 [Rank 0] time (ms) | fwd: 1429.99 | bwd: 2157.59 | bwd_inner: 2143.40 | bwd_allreduce: 13.31 | step: 28.85 2%|▏ | 67/2865 [25:24<11:43:43, 15.09s/it] {'loss': 1.151, 'learning_rate': 3.116279069767442e-05, 'epoch': 0.07} 2%|▏ | 67/2865 [25:24<11:43:43, 15.09s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 165.32 | bwd_microstep: 232.48 | bwd_inner_microstep: 232.07 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 180.07 | bwd_microstep: 278.01 | bwd_inner_microstep: 277.86 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 193.47 | bwd_microstep: 274.81 | bwd_inner_microstep: 274.64 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 212.15 | bwd_microstep: 311.26 | bwd_inner_microstep: 311.09 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 200.03 | bwd_microstep: 283.61 | bwd_inner_microstep: 283.31 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 170.62 | bwd_microstep: 264.32 | bwd_inner_microstep: 264.18 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 178.18 | bwd_microstep: 299.48 | bwd_inner_microstep: 299.15 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 190.99 | bwd_microstep: 293.86 | bwd_inner_microstep: 282.91 | bwd_allreduce_microstep: 10.76 | step_microstep: 22.03 [Rank 0] time (ms) | fwd: 1490.69 | bwd: 2237.95 | bwd_inner: 2225.62 | bwd_allreduce: 11.48 | step: 22.81 2%|▏ | 68/2865 [25:28<9:07:22, 11.74s/it] {'loss': 1.0023, 'learning_rate': 3.162790697674419e-05, 'epoch': 0.07} 2%|▏ | 68/2865 [25:28<9:07:22, 11.74s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 173.39 | bwd_microstep: 293.75 | bwd_inner_microstep: 293.50 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 186.95 | bwd_microstep: 283.81 | bwd_inner_microstep: 283.66 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 193.38 | bwd_microstep: 290.44 | bwd_inner_microstep: 289.98 | bwd_allreduce_microstep: 0.20 | step_microstep: 0.25 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 181.04 | bwd_microstep: 296.08 | bwd_inner_microstep: 295.85 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 188.47 | bwd_microstep: 325.25 | bwd_inner_microstep: 325.08 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 169.90 | bwd_microstep: 240.52 | bwd_inner_microstep: 240.37 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 176.59 | bwd_microstep: 265.20 | bwd_inner_microstep: 264.81 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.93 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 178.68 | bwd_microstep: 328.94 | bwd_inner_microstep: 316.54 | bwd_allreduce_microstep: 12.29 | step_microstep: 21.57 [Rank 0] time (ms) | fwd: 1448.25 | bwd: 2324.13 | bwd_inner: 2310.24 | bwd_allreduce: 13.03 | step: 22.50 2%|▏ | 69/2865 [25:32<7:18:41, 9.41s/it] {'loss': 1.0705, 'learning_rate': 3.2093023255813954e-05, 'epoch': 0.07} 2%|▏ | 69/2865 [25:32<7:18:41, 9.41s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 173.20 | bwd_microstep: 276.92 | bwd_inner_microstep: 276.76 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 176.81 | bwd_microstep: 280.62 | bwd_inner_microstep: 280.41 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 178.36 | bwd_microstep: 269.90 | bwd_inner_microstep: 269.67 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 167.75 | bwd_microstep: 239.64 | bwd_inner_microstep: 239.26 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 176.29 | bwd_microstep: 262.32 | bwd_inner_microstep: 262.01 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 209.66 | bwd_microstep: 273.59 | bwd_inner_microstep: 273.43 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 166.43 | bwd_microstep: 266.08 | bwd_inner_microstep: 265.90 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.10 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 166.58 | bwd_microstep: 299.04 | bwd_inner_microstep: 281.72 | bwd_allreduce_microstep: 17.17 | step_microstep: 23.59 [Rank 0] time (ms) | fwd: 1414.96 | bwd: 2168.26 | bwd_inner: 2149.58 | bwd_allreduce: 17.81 | step: 24.38 2%|▏ | 70/2865 [25:35<5:59:55, 7.73s/it] {'loss': 1.0287, 'learning_rate': 3.2558139534883724e-05, 'epoch': 0.07} 2%|▏ | 70/2865 [25:35<5:59:55, 7.73s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 185.04 | bwd_microstep: 281.53 | bwd_inner_microstep: 281.33 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 199.00 | bwd_microstep: 290.13 | bwd_inner_microstep: 289.74 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 163.15 | bwd_microstep: 318.33 | bwd_inner_microstep: 318.17 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 179.93 | bwd_microstep: 288.71 | bwd_inner_microstep: 288.42 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 162.77 | bwd_microstep: 250.35 | bwd_inner_microstep: 250.07 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 170.26 | bwd_microstep: 269.25 | bwd_inner_microstep: 269.00 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 175.94 | bwd_microstep: 239.03 | bwd_inner_microstep: 238.71 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 173.82 | bwd_microstep: 319.59 | bwd_inner_microstep: 309.73 | bwd_allreduce_microstep: 9.72 | step_microstep: 21.21 [Rank 0] time (ms) | fwd: 1409.80 | bwd: 2257.06 | bwd_inner: 2245.61 | bwd_allreduce: 10.50 | step: 22.05 2%|▏ | 71/2865 [25:39<5:05:35, 6.56s/it] {'loss': 1.3004, 'learning_rate': 3.3023255813953495e-05, 'epoch': 0.07} 2%|▏ | 71/2865 [25:39<5:05:35, 6.56s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 165.38 | bwd_microstep: 257.36 | bwd_inner_microstep: 257.16 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 161.86 | bwd_microstep: 222.97 | bwd_inner_microstep: 222.89 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 161.69 | bwd_microstep: 237.57 | bwd_inner_microstep: 237.34 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 166.61 | bwd_microstep: 225.92 | bwd_inner_microstep: 225.53 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 178.19 | bwd_microstep: 224.20 | bwd_inner_microstep: 223.84 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 168.88 | bwd_microstep: 247.35 | bwd_inner_microstep: 247.15 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 162.25 | bwd_microstep: 240.33 | bwd_inner_microstep: 240.09 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 166.91 | bwd_microstep: 273.71 | bwd_inner_microstep: 260.21 | bwd_allreduce_microstep: 13.39 | step_microstep: 21.38 [Rank 0] time (ms) | fwd: 1331.63 | bwd: 1929.52 | bwd_inner: 1914.62 | bwd_allreduce: 14.07 | step: 22.07 3%|▎ | 72/2865 [25:43<4:22:05, 5.63s/it] {'loss': 1.1513, 'learning_rate': 3.348837209302326e-05, 'epoch': 0.08} 3%|▎ | 72/2865 [25:43<4:22:05, 5.63s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 162.32 | bwd_microstep: 227.65 | bwd_inner_microstep: 227.52 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 163.90 | bwd_microstep: 228.59 | bwd_inner_microstep: 228.36 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 166.79 | bwd_microstep: 229.70 | bwd_inner_microstep: 229.53 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 165.06 | bwd_microstep: 236.71 | bwd_inner_microstep: 236.57 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 168.05 | bwd_microstep: 259.36 | bwd_inner_microstep: 259.17 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 180.51 | bwd_microstep: 276.36 | bwd_inner_microstep: 276.21 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 178.01 | bwd_microstep: 317.05 | bwd_inner_microstep: 316.85 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.14 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 156.03 | bwd_microstep: 284.65 | bwd_inner_microstep: 270.26 | bwd_allreduce_microstep: 14.27 | step_microstep: 27.11 [Rank 0] time (ms) | fwd: 1340.56 | bwd: 2060.17 | bwd_inner: 2044.78 | bwd_allreduce: 14.76 | step: 27.73 3%|▎ | 73/2865 [25:46<3:53:25, 5.02s/it] {'loss': 1.0704, 'learning_rate': 3.395348837209302e-05, 'epoch': 0.08} 3%|▎ | 73/2865 [25:46<3:53:25, 5.02s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 942 [Rank 0] time (ms) | fwd_microstep: 177.54 | bwd_microstep: 282.53 | bwd_inner_microstep: 282.28 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 189.54 | bwd_microstep: 270.23 | bwd_inner_microstep: 269.96 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 169.85 | bwd_microstep: 288.96 | bwd_inner_microstep: 288.66 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 176.45 | bwd_microstep: 264.25 | bwd_inner_microstep: 263.86 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 164.32 | bwd_microstep: 244.38 | bwd_inner_microstep: 244.15 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 192.52 | bwd_microstep: 276.33 | bwd_inner_microstep: 275.97 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 181.15 | bwd_microstep: 293.18 | bwd_inner_microstep: 292.79 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 180.98 | bwd_microstep: 288.96 | bwd_inner_microstep: 278.72 | bwd_allreduce_microstep: 10.06 | step_microstep: 21.37 [Rank 0] time (ms) | fwd: 1432.23 | bwd: 2209.00 | bwd_inner: 2196.95 | bwd_allreduce: 10.89 | step: 22.33 3%|▎ | 74/2865 [25:50<3:36:53, 4.66s/it] {'loss': 1.479, 'learning_rate': 3.441860465116279e-05, 'epoch': 0.08} 3%|▎ | 74/2865 [25:50<3:36:53, 4.66s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 938 [Rank 0] time (ms) | fwd_microstep: 181.05 | bwd_microstep: 282.88 | bwd_inner_microstep: 282.62 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 180.64 | bwd_microstep: 284.92 | bwd_inner_microstep: 284.70 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 174.07 | bwd_microstep: 273.87 | bwd_inner_microstep: 273.72 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 184.26 | bwd_microstep: 288.28 | bwd_inner_microstep: 288.12 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 167.40 | bwd_microstep: 233.62 | bwd_inner_microstep: 233.30 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 169.46 | bwd_microstep: 265.80 | bwd_inner_microstep: 265.65 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 169.97 | bwd_microstep: 281.28 | bwd_inner_microstep: 281.09 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 166.98 | bwd_microstep: 288.77 | bwd_inner_microstep: 271.04 | bwd_allreduce_microstep: 17.60 | step_microstep: 21.89 [Rank 0] time (ms) | fwd: 1393.71 | bwd: 2199.54 | bwd_inner: 2180.53 | bwd_allreduce: 18.20 | step: 22.57 3%|▎ | 75/2865 [25:54<3:24:32, 4.40s/it] {'loss': 1.0527, 'learning_rate': 3.488372093023256e-05, 'epoch': 0.08} 3%|▎ | 75/2865 [25:54<3:24:32, 4.40s/it][INFO|trainer.py:3242] 2026-05-14 20:46:37,090 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 20:46:37,090 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 20:46:37,090 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-75 [INFO|configuration_utils.py:473] 2026-05-14 20:51:44,587 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-75/config.json [INFO|configuration_utils.py:594] 2026-05-14 20:51:44,594 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-75/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 20:51:49,606 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-75/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 20:51:49,617 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-75/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 20:51:49,624 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-75/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 20:51:49,626 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-75/added_tokens.json dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 304260.37 | bwd_microstep: 246.59 | bwd_inner_microstep: 246.40 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.11 | bwd_microstep: 223.34 | bwd_inner_microstep: 222.95 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.53 | bwd_microstep: 236.81 | bwd_inner_microstep: 236.51 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 162.24 | bwd_microstep: 234.45 | bwd_inner_microstep: 234.17 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 164.89 | bwd_microstep: 221.98 | bwd_inner_microstep: 221.61 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.38 | bwd_microstep: 219.38 | bwd_inner_microstep: 219.03 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 167.23 | bwd_microstep: 230.03 | bwd_inner_microstep: 229.65 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.97 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 162.98 | bwd_microstep: 289.57 | bwd_inner_microstep: 277.14 | bwd_allreduce_microstep: 12.26 | step_microstep: 23.36 [Rank 0] time (ms) | fwd: 305389.49 | bwd: 1902.35 | bwd_inner: 1887.92 | bwd_allreduce: 13.26 | step: 24.35 3%|▎ | 76/2865 [31:19<77:57:45, 100.63s/it] {'loss': 1.1885, 'learning_rate': 3.5348837209302326e-05, 'epoch': 0.08} 3%|▎ | 76/2865 [31:19<77:57:45, 100.63s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 161.40 | bwd_microstep: 247.40 | bwd_inner_microstep: 247.10 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 162.43 | bwd_microstep: 263.80 | bwd_inner_microstep: 263.63 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 163.96 | bwd_microstep: 231.85 | bwd_inner_microstep: 231.54 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 161.96 | bwd_microstep: 232.08 | bwd_inner_microstep: 231.83 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 172.85 | bwd_microstep: 248.51 | bwd_inner_microstep: 248.18 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 168.16 | bwd_microstep: 251.04 | bwd_inner_microstep: 250.94 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 163.20 | bwd_microstep: 240.67 | bwd_inner_microstep: 240.37 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 170.13 | bwd_microstep: 272.05 | bwd_inner_microstep: 261.88 | bwd_allreduce_microstep: 10.00 | step_microstep: 21.25 [Rank 0] time (ms) | fwd: 1323.92 | bwd: 1987.51 | bwd_inner: 1975.92 | bwd_allreduce: 10.67 | step: 21.92 3%|▎ | 77/2865 [31:23<55:22:07, 71.49s/it] {'loss': 1.0932, 'learning_rate': 3.58139534883721e-05, 'epoch': 0.08} 3%|▎ | 77/2865 [31:23<55:22:07, 71.49s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.57 | bwd_microstep: 217.43 | bwd_inner_microstep: 217.31 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 160.86 | bwd_microstep: 227.00 | bwd_inner_microstep: 226.71 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.08 | bwd_microstep: 221.71 | bwd_inner_microstep: 221.31 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.51 | bwd_microstep: 219.03 | bwd_inner_microstep: 218.93 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 160.59 | bwd_microstep: 221.98 | bwd_inner_microstep: 221.57 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 146.76 | bwd_microstep: 190.09 | bwd_inner_microstep: 189.75 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 162.52 | bwd_microstep: 216.94 | bwd_inner_microstep: 216.67 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.32 | bwd_microstep: 280.62 | bwd_inner_microstep: 266.84 | bwd_allreduce_microstep: 13.60 | step_microstep: 21.78 [Rank 0] time (ms) | fwd: 1275.09 | bwd: 1794.93 | bwd_inner: 1779.53 | bwd_allreduce: 14.35 | step: 22.61 3%|▎ | 78/2865 [31:26<39:29:52, 51.02s/it] {'loss': 1.3255, 'learning_rate': 3.627906976744186e-05, 'epoch': 0.08} 3%|▎ | 78/2865 [31:26<39:29:52, 51.02s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 167.81 | bwd_microstep: 247.37 | bwd_inner_microstep: 247.09 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 177.31 | bwd_microstep: 271.25 | bwd_inner_microstep: 270.94 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 176.59 | bwd_microstep: 263.50 | bwd_inner_microstep: 263.13 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 177.87 | bwd_microstep: 260.69 | bwd_inner_microstep: 260.55 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 173.99 | bwd_microstep: 259.41 | bwd_inner_microstep: 258.98 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 174.16 | bwd_microstep: 256.09 | bwd_inner_microstep: 255.68 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 176.60 | bwd_microstep: 265.48 | bwd_inner_microstep: 265.13 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.46 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 178.71 | bwd_microstep: 350.88 | bwd_inner_microstep: 333.23 | bwd_allreduce_microstep: 17.51 | step_microstep: 30.39 [Rank 0] time (ms) | fwd: 1402.88 | bwd: 2174.86 | bwd_inner: 2155.25 | bwd_allreduce: 18.45 | step: 31.28 3%|▎ | 79/2865 [31:30<28:32:04, 36.87s/it] {'loss': 1.1993, 'learning_rate': 3.674418604651163e-05, 'epoch': 0.08} 3%|▎ | 79/2865 [31:30<28:32:04, 36.87s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 187.82 | bwd_microstep: 258.60 | bwd_inner_microstep: 258.10 | bwd_allreduce_microstep: 0.22 | step_microstep: 0.25 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 944 [Rank 0] time (ms) | fwd_microstep: 200.84 | bwd_microstep: 291.56 | bwd_inner_microstep: 291.25 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 175.66 | bwd_microstep: 308.06 | bwd_inner_microstep: 307.72 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 169.78 | bwd_microstep: 237.27 | bwd_inner_microstep: 236.88 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 173.43 | bwd_microstep: 254.70 | bwd_inner_microstep: 254.38 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 172.79 | bwd_microstep: 268.65 | bwd_inner_microstep: 268.28 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 190.82 | bwd_microstep: 267.83 | bwd_inner_microstep: 267.28 | bwd_allreduce_microstep: 0.20 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.46 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 194.37 | bwd_microstep: 306.38 | bwd_inner_microstep: 291.50 | bwd_allreduce_microstep: 14.76 | step_microstep: 24.50 [Rank 0] time (ms) | fwd: 1465.34 | bwd: 2193.28 | bwd_inner: 2176.01 | bwd_allreduce: 15.92 | step: 25.65 3%|▎ | 80/2865 [31:34<20:52:22, 26.98s/it] {'loss': 1.1773, 'learning_rate': 3.72093023255814e-05, 'epoch': 0.08} 3%|▎ | 80/2865 [31:34<20:52:22, 26.98s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 197.84 | bwd_microstep: 268.62 | bwd_inner_microstep: 268.45 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 201.13 | bwd_microstep: 269.35 | bwd_inner_microstep: 269.14 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 182.40 | bwd_microstep: 288.41 | bwd_inner_microstep: 288.26 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 185.68 | bwd_microstep: 282.91 | bwd_inner_microstep: 282.73 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 174.44 | bwd_microstep: 266.05 | bwd_inner_microstep: 265.88 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 177.34 | bwd_microstep: 257.51 | bwd_inner_microstep: 257.35 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 174.17 | bwd_microstep: 269.24 | bwd_inner_microstep: 269.10 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.03 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 169.61 | bwd_microstep: 309.12 | bwd_inner_microstep: 298.11 | bwd_allreduce_microstep: 10.93 | step_microstep: 21.65 [Rank 0] time (ms) | fwd: 1462.47 | bwd: 2211.30 | bwd_inner: 2199.26 | bwd_allreduce: 11.45 | step: 22.33 3%|▎ | 81/2865 [31:37<15:30:32, 20.05s/it] {'loss': 1.16, 'learning_rate': 3.7674418604651165e-05, 'epoch': 0.08} 3%|▎ | 81/2865 [31:37<15:30:32, 20.05s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 175.55 | bwd_microstep: 306.26 | bwd_inner_microstep: 306.01 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 184.42 | bwd_microstep: 276.29 | bwd_inner_microstep: 276.10 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 179.98 | bwd_microstep: 260.82 | bwd_inner_microstep: 260.49 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 180.12 | bwd_microstep: 267.11 | bwd_inner_microstep: 266.89 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 174.79 | bwd_microstep: 259.95 | bwd_inner_microstep: 259.64 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 173.91 | bwd_microstep: 275.61 | bwd_inner_microstep: 275.24 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 171.55 | bwd_microstep: 270.04 | bwd_inner_microstep: 269.89 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.96 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 174.47 | bwd_microstep: 308.93 | bwd_inner_microstep: 297.58 | bwd_allreduce_microstep: 11.19 | step_microstep: 22.11 [Rank 0] time (ms) | fwd: 1414.64 | bwd: 2225.11 | bwd_inner: 2212.32 | bwd_allreduce: 11.86 | step: 22.78 3%|▎ | 82/2865 [31:41<11:44:50, 15.20s/it] {'loss': 1.1716, 'learning_rate': 3.8139534883720935e-05, 'epoch': 0.09} 3%|▎ | 82/2865 [31:41<11:44:50, 15.20s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 171.80 | bwd_microstep: 266.19 | bwd_inner_microstep: 265.81 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 180.76 | bwd_microstep: 282.12 | bwd_inner_microstep: 281.71 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.27 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 181.75 | bwd_microstep: 285.77 | bwd_inner_microstep: 285.36 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.26 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 182.26 | bwd_microstep: 278.31 | bwd_inner_microstep: 278.09 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 184.43 | bwd_microstep: 289.75 | bwd_inner_microstep: 289.58 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 183.13 | bwd_microstep: 300.36 | bwd_inner_microstep: 300.17 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 182.69 | bwd_microstep: 302.96 | bwd_inner_microstep: 302.82 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.00 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 165.17 | bwd_microstep: 277.84 | bwd_inner_microstep: 267.58 | bwd_allreduce_microstep: 10.11 | step_microstep: 22.03 [Rank 0] time (ms) | fwd: 1431.86 | bwd: 2283.47 | bwd_inner: 2271.58 | bwd_allreduce: 10.93 | step: 23.18 3%|▎ | 83/2865 [31:45<9:07:52, 11.82s/it] {'loss': 1.3089, 'learning_rate': 3.8604651162790706e-05, 'epoch': 0.09} 3%|▎ | 83/2865 [31:45<9:07:52, 11.82s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 185.63 | bwd_microstep: 254.04 | bwd_inner_microstep: 253.87 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 167.14 | bwd_microstep: 232.61 | bwd_inner_microstep: 232.49 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 164.89 | bwd_microstep: 234.51 | bwd_inner_microstep: 234.27 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 175.63 | bwd_microstep: 233.57 | bwd_inner_microstep: 233.48 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 162.28 | bwd_microstep: 232.40 | bwd_inner_microstep: 232.23 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 166.79 | bwd_microstep: 240.66 | bwd_inner_microstep: 240.53 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 151.10 | bwd_microstep: 211.57 | bwd_inner_microstep: 211.40 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 166.06 | bwd_microstep: 270.48 | bwd_inner_microstep: 260.61 | bwd_allreduce_microstep: 9.78 | step_microstep: 21.05 [Rank 0] time (ms) | fwd: 1339.39 | bwd: 1909.96 | bwd_inner: 1899.07 | bwd_allreduce: 10.31 | step: 21.68 3%|▎ | 84/2865 [31:49<7:11:12, 9.30s/it] {'loss': 1.0025, 'learning_rate': 3.906976744186047e-05, 'epoch': 0.09} 3%|▎ | 84/2865 [31:49<7:11:12, 9.30s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 165.49 | bwd_microstep: 228.56 | bwd_inner_microstep: 228.32 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 165.48 | bwd_microstep: 229.47 | bwd_inner_microstep: 229.27 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 166.84 | bwd_microstep: 229.88 | bwd_inner_microstep: 229.67 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 168.92 | bwd_microstep: 230.89 | bwd_inner_microstep: 230.70 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 163.16 | bwd_microstep: 234.77 | bwd_inner_microstep: 234.63 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 166.75 | bwd_microstep: 232.89 | bwd_inner_microstep: 232.72 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 167.00 | bwd_microstep: 249.18 | bwd_inner_microstep: 248.67 | bwd_allreduce_microstep: 0.23 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 165.30 | bwd_microstep: 267.40 | bwd_inner_microstep: 257.33 | bwd_allreduce_microstep: 9.99 | step_microstep: 21.10 [Rank 0] time (ms) | fwd: 1328.81 | bwd: 1903.19 | bwd_inner: 1891.60 | bwd_allreduce: 10.77 | step: 21.82 3%|▎ | 85/2865 [31:52<5:49:19, 7.54s/it] {'loss': 1.1443, 'learning_rate': 3.953488372093023e-05, 'epoch': 0.09} 3%|▎ | 85/2865 [31:52<5:49:19, 7.54s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 167.26 | bwd_microstep: 239.79 | bwd_inner_microstep: 239.65 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 168.84 | bwd_microstep: 233.23 | bwd_inner_microstep: 233.13 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 162.29 | bwd_microstep: 234.78 | bwd_inner_microstep: 234.65 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 177.06 | bwd_microstep: 232.49 | bwd_inner_microstep: 232.32 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 180.67 | bwd_microstep: 258.71 | bwd_inner_microstep: 258.61 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 163.95 | bwd_microstep: 246.47 | bwd_inner_microstep: 246.12 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 167.95 | bwd_microstep: 240.12 | bwd_inner_microstep: 239.91 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.93 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 168.08 | bwd_microstep: 286.03 | bwd_inner_microstep: 275.87 | bwd_allreduce_microstep: 10.07 | step_microstep: 21.41 [Rank 0] time (ms) | fwd: 1355.96 | bwd: 1971.72 | bwd_inner: 1960.53 | bwd_allreduce: 10.58 | step: 22.03 3%|▎ | 86/2865 [31:56<4:53:08, 6.33s/it] {'loss': 1.0572, 'learning_rate': 4e-05, 'epoch': 0.09} 3%|▎ | 86/2865 [31:56<4:53:08, 6.33s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 938 [Rank 0] time (ms) | fwd_microstep: 170.90 | bwd_microstep: 235.37 | bwd_inner_microstep: 235.24 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 165.09 | bwd_microstep: 226.58 | bwd_inner_microstep: 226.43 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 173.86 | bwd_microstep: 228.88 | bwd_inner_microstep: 228.77 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 175.90 | bwd_microstep: 231.41 | bwd_inner_microstep: 231.29 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 164.31 | bwd_microstep: 227.26 | bwd_inner_microstep: 226.98 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 168.88 | bwd_microstep: 230.61 | bwd_inner_microstep: 230.39 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 166.00 | bwd_microstep: 232.21 | bwd_inner_microstep: 231.83 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.96 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 182.90 | bwd_microstep: 276.27 | bwd_inner_microstep: 266.42 | bwd_allreduce_microstep: 9.77 | step_microstep: 24.13 [Rank 0] time (ms) | fwd: 1367.72 | bwd: 1888.72 | bwd_inner: 1877.55 | bwd_allreduce: 10.43 | step: 24.94 3%|▎ | 87/2865 [31:59<4:12:53, 5.46s/it] {'loss': 1.14, 'learning_rate': 3.999998722024272e-05, 'epoch': 0.09} 3%|▎ | 87/2865 [31:59<4:12:53, 5.46s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 169.26 | bwd_microstep: 227.81 | bwd_inner_microstep: 227.42 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 166.59 | bwd_microstep: 250.54 | bwd_inner_microstep: 250.42 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 168.12 | bwd_microstep: 230.46 | bwd_inner_microstep: 230.28 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 164.64 | bwd_microstep: 229.84 | bwd_inner_microstep: 229.70 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 164.83 | bwd_microstep: 247.90 | bwd_inner_microstep: 247.79 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.13 | bwd_microstep: 249.57 | bwd_inner_microstep: 249.45 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 166.98 | bwd_microstep: 246.21 | bwd_inner_microstep: 246.08 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 165.20 | bwd_microstep: 271.30 | bwd_inner_microstep: 261.11 | bwd_allreduce_microstep: 10.10 | step_microstep: 20.99 [Rank 0] time (ms) | fwd: 1325.64 | bwd: 1953.73 | bwd_inner: 1942.46 | bwd_allreduce: 10.64 | step: 21.66 3%|▎ | 88/2865 [32:02<3:44:54, 4.86s/it] {'loss': 1.1048, 'learning_rate': 3.999994888098721e-05, 'epoch': 0.09} 3%|▎ | 88/2865 [32:02<3:44:54, 4.86s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 166.73 | bwd_microstep: 227.80 | bwd_inner_microstep: 227.62 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 167.54 | bwd_microstep: 229.94 | bwd_inner_microstep: 229.83 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 167.30 | bwd_microstep: 235.38 | bwd_inner_microstep: 235.26 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.50 | bwd_microstep: 236.95 | bwd_inner_microstep: 236.82 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 167.77 | bwd_microstep: 228.21 | bwd_inner_microstep: 228.09 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 165.09 | bwd_microstep: 246.29 | bwd_inner_microstep: 246.17 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 150.47 | bwd_microstep: 204.51 | bwd_inner_microstep: 204.36 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 168.44 | bwd_microstep: 265.42 | bwd_inner_microstep: 256.61 | bwd_allreduce_microstep: 8.73 | step_microstep: 21.29 [Rank 0] time (ms) | fwd: 1314.71 | bwd: 1874.60 | bwd_inner: 1864.91 | bwd_allreduce: 9.18 | step: 21.84 3%|▎ | 89/2865 [32:06<3:24:04, 4.41s/it] {'loss': 1.5029, 'learning_rate': 3.9999884982282465e-05, 'epoch': 0.09} 3%|▎ | 89/2865 [32:06<3:24:04, 4.41s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 158.17 | bwd_microstep: 231.52 | bwd_inner_microstep: 231.35 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 159.30 | bwd_microstep: 234.70 | bwd_inner_microstep: 234.51 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 167.55 | bwd_microstep: 242.85 | bwd_inner_microstep: 242.65 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 164.91 | bwd_microstep: 228.87 | bwd_inner_microstep: 228.76 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 165.85 | bwd_microstep: 240.83 | bwd_inner_microstep: 240.70 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 178.44 | bwd_microstep: 271.59 | bwd_inner_microstep: 271.42 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 167.69 | bwd_microstep: 249.39 | bwd_inner_microstep: 249.26 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.94 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 167.31 | bwd_microstep: 313.95 | bwd_inner_microstep: 303.77 | bwd_allreduce_microstep: 10.09 | step_microstep: 21.12 [Rank 0] time (ms) | fwd: 1329.10 | bwd: 2013.80 | bwd_inner: 2002.63 | bwd_allreduce: 10.60 | step: 21.72 3%|▎ | 90/2865 [32:09<3:11:43, 4.15s/it] {'loss': 1.0508, 'learning_rate': 3.9999795524210154e-05, 'epoch': 0.09} 3%|▎ | 90/2865 [32:09<3:11:43, 4.15s/it][INFO|trainer.py:3242] 2026-05-14 20:52:52,534 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 20:52:52,534 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 20:52:52,534 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-90 [INFO|configuration_utils.py:473] 2026-05-14 20:57:59,548 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-90/config.json [INFO|configuration_utils.py:594] 2026-05-14 20:57:59,556 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-90/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 20:58:05,036 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-90/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 20:58:05,046 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-90/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 20:58:05,052 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-90/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 20:58:05,054 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-90/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 20:58:14,455 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-15] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 303933.08 | bwd_microstep: 239.95 | bwd_inner_microstep: 239.61 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 160.55 | bwd_microstep: 261.87 | bwd_inner_microstep: 261.50 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.39 | bwd_microstep: 232.56 | bwd_inner_microstep: 232.18 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.78 | bwd_microstep: 221.81 | bwd_inner_microstep: 221.52 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 171.37 | bwd_microstep: 217.38 | bwd_inner_microstep: 217.12 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 168.10 | bwd_microstep: 216.36 | bwd_inner_microstep: 216.17 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 159.92 | bwd_microstep: 216.61 | bwd_inner_microstep: 216.38 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.01 | optimizer_gradients: 2.91 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 172.44 | bwd_microstep: 249.11 | bwd_inner_microstep: 238.84 | bwd_allreduce_microstep: 10.18 | step_microstep: 21.42 [Rank 0] time (ms) | fwd: 305075.46 | bwd: 1855.81 | bwd_inner: 1843.77 | bwd_allreduce: 11.03 | step: 22.08 3%|▎ | 91/2865 [37:35<77:30:05, 100.58s/it] {'loss': 1.0417, 'learning_rate': 3.999968050688459e-05, 'epoch': 0.1} 3%|▎ | 91/2865 [37:35<77:30:05, 100.58s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 161.74 | bwd_microstep: 220.57 | bwd_inner_microstep: 220.30 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 161.96 | bwd_microstep: 219.49 | bwd_inner_microstep: 219.35 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 162.79 | bwd_microstep: 225.89 | bwd_inner_microstep: 225.48 | bwd_allreduce_microstep: 0.20 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.59 | bwd_microstep: 238.93 | bwd_inner_microstep: 238.53 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.34 | bwd_microstep: 235.10 | bwd_inner_microstep: 234.80 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 146.89 | bwd_microstep: 199.59 | bwd_inner_microstep: 199.31 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 160.56 | bwd_microstep: 224.76 | bwd_inner_microstep: 224.38 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 161.65 | bwd_microstep: 250.35 | bwd_inner_microstep: 240.36 | bwd_allreduce_microstep: 9.83 | step_microstep: 21.15 [Rank 0] time (ms) | fwd: 1278.40 | bwd: 1814.82 | bwd_inner: 1803.00 | bwd_allreduce: 10.67 | step: 22.11 3%|▎ | 92/2865 [37:38<54:59:06, 71.38s/it] {'loss': 1.0267, 'learning_rate': 3.9999539930452766e-05, 'epoch': 0.1} 3%|▎ | 92/2865 [37:38<54:59:06, 71.38s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 942 [Rank 0] time (ms) | fwd_microstep: 162.19 | bwd_microstep: 244.53 | bwd_inner_microstep: 244.13 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 163.71 | bwd_microstep: 218.44 | bwd_inner_microstep: 218.31 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 161.69 | bwd_microstep: 219.16 | bwd_inner_microstep: 218.72 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.31 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.86 | bwd_microstep: 218.31 | bwd_inner_microstep: 217.92 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 162.03 | bwd_microstep: 220.55 | bwd_inner_microstep: 220.47 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 161.85 | bwd_microstep: 216.35 | bwd_inner_microstep: 216.22 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 163.30 | bwd_microstep: 219.17 | bwd_inner_microstep: 218.99 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 161.38 | bwd_microstep: 228.26 | bwd_inner_microstep: 218.28 | bwd_allreduce_microstep: 9.82 | step_microstep: 21.80 [Rank 0] time (ms) | fwd: 1297.88 | bwd: 1784.95 | bwd_inner: 1773.42 | bwd_allreduce: 10.59 | step: 22.69 3%|▎ | 93/2865 [37:41<39:13:55, 50.95s/it] {'loss': 1.035, 'learning_rate': 3.999937379509434e-05, 'epoch': 0.1} 3%|▎ | 93/2865 [37:41<39:13:55, 50.95s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 163.08 | bwd_microstep: 204.79 | bwd_inner_microstep: 204.43 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 163.82 | bwd_microstep: 204.80 | bwd_inner_microstep: 204.36 | bwd_allreduce_microstep: 0.20 | step_microstep: 0.24 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 163.67 | bwd_microstep: 198.38 | bwd_inner_microstep: 197.82 | bwd_allreduce_microstep: 0.31 | step_microstep: 0.24 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 165.23 | bwd_microstep: 200.65 | bwd_inner_microstep: 200.43 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 169.23 | bwd_microstep: 199.13 | bwd_inner_microstep: 199.03 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 163.45 | bwd_microstep: 200.80 | bwd_inner_microstep: 200.51 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 163.78 | bwd_microstep: 204.16 | bwd_inner_microstep: 203.67 | bwd_allreduce_microstep: 0.25 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.86 [Rank 0] time (ms) | fwd_microstep: 164.16 | bwd_microstep: 233.72 | bwd_inner_microstep: 223.01 | bwd_allreduce_microstep: 10.60 | step_microstep: 21.57 [Rank 0] time (ms) | fwd: 1316.31 | bwd: 1646.67 | bwd_inner: 1633.79 | bwd_allreduce: 11.60 | step: 22.59 3%|▎ | 94/2865 [37:45<28:10:47, 36.61s/it] {'loss': 1.1249, 'learning_rate': 3.999918210102163e-05, 'epoch': 0.1} 3%|▎ | 94/2865 [37:45<28:10:47, 36.61s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 942 [Rank 0] time (ms) | fwd_microstep: 163.37 | bwd_microstep: 195.24 | bwd_inner_microstep: 195.04 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 161.71 | bwd_microstep: 190.23 | bwd_inner_microstep: 190.17 | bwd_allreduce_microstep: 0.02 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 164.63 | bwd_microstep: 189.74 | bwd_inner_microstep: 189.51 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.53 | bwd_microstep: 192.86 | bwd_inner_microstep: 192.74 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 162.08 | bwd_microstep: 194.27 | bwd_inner_microstep: 193.91 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.13 | bwd_microstep: 238.27 | bwd_inner_microstep: 238.14 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 163.37 | bwd_microstep: 221.12 | bwd_inner_microstep: 220.78 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.09 | bwd_microstep: 252.24 | bwd_inner_microstep: 242.63 | bwd_allreduce_microstep: 9.52 | step_microstep: 21.13 [Rank 0] time (ms) | fwd: 1298.81 | bwd: 1674.07 | bwd_inner: 1663.22 | bwd_allreduce: 10.14 | step: 21.80 3%|▎ | 95/2865 [37:48<20:26:38, 26.57s/it] {'loss': 1.2173, 'learning_rate': 3.999896484847961e-05, 'epoch': 0.1} 3%|▎ | 95/2865 [37:48<20:26:38, 26.57s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.02 | bwd_microstep: 217.89 | bwd_inner_microstep: 217.53 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 161.34 | bwd_microstep: 220.37 | bwd_inner_microstep: 220.23 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 146.97 | bwd_microstep: 193.62 | bwd_inner_microstep: 193.41 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.60 | bwd_microstep: 224.68 | bwd_inner_microstep: 224.55 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.03 | bwd_microstep: 225.99 | bwd_inner_microstep: 225.86 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 161.19 | bwd_microstep: 231.80 | bwd_inner_microstep: 231.66 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 161.07 | bwd_microstep: 222.05 | bwd_inner_microstep: 221.88 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | optimizer_allgather: 0.03 | optimizer_gradients: 2.94 | optimizer_step: 4.91 [Rank 0] time (ms) | fwd_microstep: 160.77 | bwd_microstep: 254.23 | bwd_inner_microstep: 244.25 | bwd_allreduce_microstep: 9.81 | step_microstep: 21.65 [Rank 0] time (ms) | fwd: 1273.88 | bwd: 1790.76 | bwd_inner: 1779.71 | bwd_allreduce: 10.33 | step: 22.25 3%|▎ | 96/2865 [37:51<15:03:02, 19.57s/it] {'loss': 0.9216, 'learning_rate': 3.999872203774593e-05, 'epoch': 0.1} 3%|▎ | 96/2865 [37:51<15:03:02, 19.57s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 946 [Rank 0] time (ms) | fwd_microstep: 162.65 | bwd_microstep: 227.37 | bwd_inner_microstep: 226.99 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 162.86 | bwd_microstep: 233.42 | bwd_inner_microstep: 233.30 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 171.85 | bwd_microstep: 216.75 | bwd_inner_microstep: 216.23 | bwd_allreduce_microstep: 0.22 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 163.78 | bwd_microstep: 212.26 | bwd_inner_microstep: 211.96 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.27 | bwd_microstep: 191.30 | bwd_inner_microstep: 190.93 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 161.15 | bwd_microstep: 193.11 | bwd_inner_microstep: 192.98 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 168.04 | bwd_microstep: 198.19 | bwd_inner_microstep: 197.92 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.93 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 168.31 | bwd_microstep: 225.55 | bwd_inner_microstep: 213.16 | bwd_allreduce_microstep: 12.21 | step_microstep: 22.45 [Rank 0] time (ms) | fwd: 1319.77 | bwd: 1698.14 | bwd_inner: 1684.02 | bwd_allreduce: 12.97 | step: 23.16 3%|▎ | 97/2865 [37:54<11:16:30, 14.66s/it] {'loss': 1.3556, 'learning_rate': 3.9998453669130886e-05, 'epoch': 0.1} 3%|▎ | 97/2865 [37:54<11:16:30, 14.66s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 941 [Rank 0] time (ms) | fwd_microstep: 166.31 | bwd_microstep: 201.36 | bwd_inner_microstep: 201.23 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 165.05 | bwd_microstep: 200.58 | bwd_inner_microstep: 200.26 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 162.27 | bwd_microstep: 196.09 | bwd_inner_microstep: 195.89 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.98 | bwd_microstep: 203.94 | bwd_inner_microstep: 203.74 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 173.27 | bwd_microstep: 199.31 | bwd_inner_microstep: 198.96 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 164.64 | bwd_microstep: 250.01 | bwd_inner_microstep: 249.75 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 173.51 | bwd_microstep: 225.86 | bwd_inner_microstep: 225.77 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 164.96 | bwd_microstep: 256.58 | bwd_inner_microstep: 246.33 | bwd_allreduce_microstep: 10.18 | step_microstep: 21.48 [Rank 0] time (ms) | fwd: 1331.86 | bwd: 1733.84 | bwd_inner: 1722.22 | bwd_allreduce: 10.81 | step: 22.09 3%|▎ | 98/2865 [37:57<8:38:28, 11.24s/it] {'loss': 1.1525, 'learning_rate': 3.999815974297746e-05, 'epoch': 0.1} 3%|▎ | 98/2865 [37:57<8:38:28, 11.24s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 164.41 | bwd_microstep: 219.19 | bwd_inner_microstep: 218.82 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 163.95 | bwd_microstep: 246.67 | bwd_inner_microstep: 246.32 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 173.63 | bwd_microstep: 222.45 | bwd_inner_microstep: 222.07 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.24 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 164.60 | bwd_microstep: 219.56 | bwd_inner_microstep: 219.23 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 163.89 | bwd_microstep: 224.54 | bwd_inner_microstep: 224.01 | bwd_allreduce_microstep: 0.20 | step_microstep: 0.24 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 169.44 | bwd_microstep: 225.79 | bwd_inner_microstep: 225.51 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.10 | bwd_microstep: 229.14 | bwd_inner_microstep: 229.03 | bwd_allreduce_microstep: 0.03 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 166.34 | bwd_microstep: 260.34 | bwd_inner_microstep: 250.25 | bwd_allreduce_microstep: 9.93 | step_microstep: 21.27 [Rank 0] time (ms) | fwd: 1329.24 | bwd: 1847.86 | bwd_inner: 1835.81 | bwd_allreduce: 10.87 | step: 22.42 3%|▎ | 99/2865 [38:01<6:49:20, 8.88s/it] {'loss': 1.1309, 'learning_rate': 3.999784025966127e-05, 'epoch': 0.1} 3%|▎ | 99/2865 [38:01<6:49:20, 8.88s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 166.65 | bwd_microstep: 224.94 | bwd_inner_microstep: 224.50 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 164.41 | bwd_microstep: 245.42 | bwd_inner_microstep: 245.17 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 170.60 | bwd_microstep: 246.57 | bwd_inner_microstep: 246.28 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 168.79 | bwd_microstep: 244.04 | bwd_inner_microstep: 243.85 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 162.42 | bwd_microstep: 243.08 | bwd_inner_microstep: 242.77 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 161.88 | bwd_microstep: 244.68 | bwd_inner_microstep: 244.48 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 162.16 | bwd_microstep: 230.33 | bwd_inner_microstep: 230.00 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.90 | bwd_microstep: 275.06 | bwd_inner_microstep: 264.42 | bwd_allreduce_microstep: 10.48 | step_microstep: 21.36 [Rank 0] time (ms) | fwd: 1319.69 | bwd: 1954.24 | bwd_inner: 1941.95 | bwd_allreduce: 11.27 | step: 22.10 3%|▎ | 100/2865 [38:04<5:34:17, 7.25s/it] {'loss': 1.0473, 'learning_rate': 3.999749521959062e-05, 'epoch': 0.1} 3%|▎ | 100/2865 [38:04<5:34:17, 7.25s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 162.92 | bwd_microstep: 224.76 | bwd_inner_microstep: 224.45 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 167.24 | bwd_microstep: 250.95 | bwd_inner_microstep: 250.74 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 171.18 | bwd_microstep: 229.45 | bwd_inner_microstep: 229.07 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 164.52 | bwd_microstep: 221.70 | bwd_inner_microstep: 221.56 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 166.82 | bwd_microstep: 247.06 | bwd_inner_microstep: 246.84 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 164.78 | bwd_microstep: 221.62 | bwd_inner_microstep: 221.26 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 171.19 | bwd_microstep: 234.63 | bwd_inner_microstep: 234.26 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.94 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 170.90 | bwd_microstep: 289.38 | bwd_inner_microstep: 279.48 | bwd_allreduce_microstep: 9.74 | step_microstep: 21.70 [Rank 0] time (ms) | fwd: 1339.41 | bwd: 1919.68 | bwd_inner: 1908.19 | bwd_allreduce: 10.46 | step: 22.47 4%|▎ | 101/2865 [38:08<4:41:30, 6.11s/it] {'loss': 1.1397, 'learning_rate': 3.999712462320645e-05, 'epoch': 0.11} 4%|▎ | 101/2865 [38:08<4:41:30, 6.11s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 938 [Rank 0] time (ms) | fwd_microstep: 166.07 | bwd_microstep: 246.82 | bwd_inner_microstep: 246.65 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 168.56 | bwd_microstep: 224.07 | bwd_inner_microstep: 223.87 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 162.27 | bwd_microstep: 220.86 | bwd_inner_microstep: 220.46 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 169.03 | bwd_microstep: 222.66 | bwd_inner_microstep: 222.41 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 163.09 | bwd_microstep: 221.86 | bwd_inner_microstep: 221.73 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 165.25 | bwd_microstep: 226.51 | bwd_inner_microstep: 225.93 | bwd_allreduce_microstep: 0.25 | step_microstep: 0.27 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 163.59 | bwd_microstep: 228.45 | bwd_inner_microstep: 228.25 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.61 | bwd_microstep: 256.80 | bwd_inner_microstep: 246.89 | bwd_allreduce_microstep: 9.83 | step_microstep: 21.21 [Rank 0] time (ms) | fwd: 1321.33 | bwd: 1848.21 | bwd_inner: 1836.67 | bwd_allreduce: 10.56 | step: 22.13 4%|▎ | 102/2865 [38:11<4:03:54, 5.30s/it] {'loss': 0.881, 'learning_rate': 3.999672847098238e-05, 'epoch': 0.11} 4%|▎ | 102/2865 [38:11<4:03:54, 5.30s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 168.11 | bwd_microstep: 252.67 | bwd_inner_microstep: 252.48 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 176.26 | bwd_microstep: 238.97 | bwd_inner_microstep: 238.58 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 163.19 | bwd_microstep: 239.67 | bwd_inner_microstep: 239.37 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.89 | bwd_microstep: 219.73 | bwd_inner_microstep: 219.39 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 161.21 | bwd_microstep: 217.22 | bwd_inner_microstep: 216.86 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 167.27 | bwd_microstep: 220.33 | bwd_inner_microstep: 220.20 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 167.85 | bwd_microstep: 225.99 | bwd_inner_microstep: 225.70 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 162.10 | bwd_microstep: 257.79 | bwd_inner_microstep: 245.79 | bwd_allreduce_microstep: 11.86 | step_microstep: 21.67 [Rank 0] time (ms) | fwd: 1327.78 | bwd: 1872.51 | bwd_inner: 1858.82 | bwd_allreduce: 12.69 | step: 22.42 4%|▎ | 103/2865 [38:15<3:37:18, 4.72s/it] {'loss': 1.2838, 'learning_rate': 3.999630676342468e-05, 'epoch': 0.11} 4%|▎ | 103/2865 [38:15<3:37:18, 4.72s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 163.53 | bwd_microstep: 221.18 | bwd_inner_microstep: 220.88 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 938 [Rank 0] time (ms) | fwd_microstep: 167.83 | bwd_microstep: 220.65 | bwd_inner_microstep: 220.35 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 164.49 | bwd_microstep: 216.76 | bwd_inner_microstep: 216.40 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 172.20 | bwd_microstep: 218.12 | bwd_inner_microstep: 217.97 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 165.59 | bwd_microstep: 224.98 | bwd_inner_microstep: 224.61 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 171.85 | bwd_microstep: 222.78 | bwd_inner_microstep: 222.54 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 170.39 | bwd_microstep: 243.49 | bwd_inner_microstep: 243.36 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 163.68 | bwd_microstep: 295.65 | bwd_inner_microstep: 283.75 | bwd_allreduce_microstep: 11.81 | step_microstep: 21.66 [Rank 0] time (ms) | fwd: 1339.42 | bwd: 1863.75 | bwd_inner: 1850.25 | bwd_allreduce: 12.53 | step: 22.46 4%|▎ | 104/2865 [38:18<3:19:28, 4.33s/it] {'loss': 1.1335, 'learning_rate': 3.9995859501072297e-05, 'epoch': 0.11} 4%|▎ | 104/2865 [38:18<3:19:28, 4.33s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 942 [Rank 0] time (ms) | fwd_microstep: 171.57 | bwd_microstep: 232.60 | bwd_inner_microstep: 232.26 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 162.12 | bwd_microstep: 220.64 | bwd_inner_microstep: 220.14 | bwd_allreduce_microstep: 0.23 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 173.56 | bwd_microstep: 222.17 | bwd_inner_microstep: 221.58 | bwd_allreduce_microstep: 0.23 | step_microstep: 0.29 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 165.67 | bwd_microstep: 227.38 | bwd_inner_microstep: 227.01 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 164.41 | bwd_microstep: 242.06 | bwd_inner_microstep: 241.86 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 147.73 | bwd_microstep: 204.05 | bwd_inner_microstep: 203.92 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 909 [Rank 0] time (ms) | fwd_microstep: 164.30 | bwd_microstep: 231.39 | bwd_inner_microstep: 231.03 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.24 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 162.23 | bwd_microstep: 265.27 | bwd_inner_microstep: 255.34 | bwd_allreduce_microstep: 9.84 | step_microstep: 24.44 [Rank 0] time (ms) | fwd: 1311.46 | bwd: 1845.79 | bwd_inner: 1833.64 | bwd_allreduce: 10.91 | step: 25.36 4%|▎ | 105/2865 [38:21<3:05:53, 4.04s/it] {'loss': 1.0985, 'learning_rate': 3.99953866844968e-05, 'epoch': 0.11} 4%|▎ | 105/2865 [38:21<3:05:53, 4.04s/it][INFO|trainer.py:3242] 2026-05-14 20:59:04,496 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 20:59:04,496 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 20:59:04,496 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-105 [INFO|configuration_utils.py:473] 2026-05-14 21:04:12,621 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-105/config.json [INFO|configuration_utils.py:594] 2026-05-14 21:04:12,629 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-105/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 21:04:16,934 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-105/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 21:04:16,945 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-105/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 21:04:16,951 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-105/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 21:04:16,953 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-105/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 21:04:23,938 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-30] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 304647.06 | bwd_microstep: 242.23 | bwd_inner_microstep: 241.89 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 164.01 | bwd_microstep: 251.16 | bwd_inner_microstep: 251.03 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 167.66 | bwd_microstep: 235.88 | bwd_inner_microstep: 235.60 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 171.49 | bwd_microstep: 248.76 | bwd_inner_microstep: 248.43 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 164.85 | bwd_microstep: 241.47 | bwd_inner_microstep: 241.21 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 162.59 | bwd_microstep: 244.90 | bwd_inner_microstep: 244.73 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 160.07 | bwd_microstep: 246.04 | bwd_inner_microstep: 245.81 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.37 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 156.40 | bwd_microstep: 270.20 | bwd_inner_microstep: 260.61 | bwd_allreduce_microstep: 9.51 | step_microstep: 22.69 [Rank 0] time (ms) | fwd: 305782.32 | bwd: 1980.79 | bwd_inner: 1969.63 | bwd_allreduce: 10.29 | step: 23.47 4%|▎ | 106/2865 [43:45<76:28:38, 99.79s/it] {'loss': 1.1491, 'learning_rate': 3.999488831430245e-05, 'epoch': 0.11} 4%|▎ | 106/2865 [43:45<76:28:38, 99.79s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 161.10 | bwd_microstep: 233.40 | bwd_inner_microstep: 233.21 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 180.91 | bwd_microstep: 238.84 | bwd_inner_microstep: 238.66 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 171.77 | bwd_microstep: 237.17 | bwd_inner_microstep: 237.03 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 169.36 | bwd_microstep: 243.25 | bwd_inner_microstep: 242.96 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 175.60 | bwd_microstep: 280.33 | bwd_inner_microstep: 280.17 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 166.74 | bwd_microstep: 252.72 | bwd_inner_microstep: 252.36 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 198.60 | bwd_microstep: 284.83 | bwd_inner_microstep: 284.62 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 167.49 | bwd_microstep: 277.47 | bwd_inner_microstep: 267.06 | bwd_allreduce_microstep: 10.32 | step_microstep: 21.10 [Rank 0] time (ms) | fwd: 1391.46 | bwd: 2048.11 | bwd_inner: 2036.38 | bwd_allreduce: 10.95 | step: 21.72 4%|▎ | 107/2865 [43:48<54:20:42, 70.94s/it] {'loss': 0.7817, 'learning_rate': 3.999436439112614e-05, 'epoch': 0.11} 4%|▎ | 107/2865 [43:48<54:20:42, 70.94s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 165.17 | bwd_microstep: 252.28 | bwd_inner_microstep: 252.09 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 179.53 | bwd_microstep: 268.72 | bwd_inner_microstep: 268.35 | bwd_allreduce_microstep: 0.20 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 172.45 | bwd_microstep: 258.33 | bwd_inner_microstep: 257.92 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 172.40 | bwd_microstep: 244.73 | bwd_inner_microstep: 244.41 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 167.08 | bwd_microstep: 337.87 | bwd_inner_microstep: 337.71 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 173.38 | bwd_microstep: 259.13 | bwd_inner_microstep: 258.71 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 170.89 | bwd_microstep: 269.83 | bwd_inner_microstep: 269.64 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 169.12 | bwd_microstep: 289.03 | bwd_inner_microstep: 278.08 | bwd_allreduce_microstep: 10.85 | step_microstep: 21.77 [Rank 0] time (ms) | fwd: 1369.91 | bwd: 2180.02 | bwd_inner: 2167.37 | bwd_allreduce: 11.68 | step: 22.53 4%|▍ | 108/2865 [43:52<38:53:10, 50.78s/it] {'loss': 1.1253, 'learning_rate': 3.9993814915637454e-05, 'epoch': 0.11} 4%|▍ | 108/2865 [43:52<38:53:10, 50.78s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 172.55 | bwd_microstep: 263.38 | bwd_inner_microstep: 263.25 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 166.67 | bwd_microstep: 257.59 | bwd_inner_microstep: 257.40 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 177.80 | bwd_microstep: 245.77 | bwd_inner_microstep: 245.37 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 166.13 | bwd_microstep: 249.21 | bwd_inner_microstep: 249.06 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 174.92 | bwd_microstep: 251.23 | bwd_inner_microstep: 251.05 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 166.05 | bwd_microstep: 241.33 | bwd_inner_microstep: 241.19 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 179.61 | bwd_microstep: 248.05 | bwd_inner_microstep: 247.77 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 172.68 | bwd_microstep: 280.18 | bwd_inner_microstep: 269.44 | bwd_allreduce_microstep: 10.53 | step_microstep: 21.46 [Rank 0] time (ms) | fwd: 1376.29 | bwd: 2036.87 | bwd_inner: 2024.90 | bwd_allreduce: 11.16 | step: 22.19 4%|▍ | 109/2865 [43:55<28:02:27, 36.63s/it] {'loss': 0.9584, 'learning_rate': 3.999323988853858e-05, 'epoch': 0.11} 4%|▍ | 109/2865 [43:55<28:02:27, 36.63s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 166.25 | bwd_microstep: 249.84 | bwd_inner_microstep: 249.70 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 165.65 | bwd_microstep: 235.86 | bwd_inner_microstep: 235.52 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 183.69 | bwd_microstep: 243.15 | bwd_inner_microstep: 242.97 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 177.66 | bwd_microstep: 255.74 | bwd_inner_microstep: 255.57 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 173.92 | bwd_microstep: 252.94 | bwd_inner_microstep: 252.81 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 168.18 | bwd_microstep: 251.79 | bwd_inner_microstep: 251.58 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.85 | bwd_microstep: 259.74 | bwd_inner_microstep: 259.53 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.97 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 175.37 | bwd_microstep: 287.41 | bwd_inner_microstep: 276.94 | bwd_allreduce_microstep: 10.37 | step_microstep: 21.29 [Rank 0] time (ms) | fwd: 1373.46 | bwd: 2036.57 | bwd_inner: 2024.89 | bwd_allreduce: 10.95 | step: 21.95 4%|▍ | 110/2865 [43:59<20:26:51, 26.72s/it] {'loss': 0.9343, 'learning_rate': 3.9992639310564414e-05, 'epoch': 0.12} 4%|▍ | 110/2865 [43:59<20:26:51, 26.72s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 170.14 | bwd_microstep: 257.57 | bwd_inner_microstep: 257.42 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 178.14 | bwd_microstep: 252.80 | bwd_inner_microstep: 252.63 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 171.35 | bwd_microstep: 264.66 | bwd_inner_microstep: 264.51 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 185.17 | bwd_microstep: 280.10 | bwd_inner_microstep: 279.93 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 165.51 | bwd_microstep: 265.40 | bwd_inner_microstep: 265.15 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 175.51 | bwd_microstep: 251.23 | bwd_inner_microstep: 251.07 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 170.29 | bwd_microstep: 280.86 | bwd_inner_microstep: 280.72 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.93 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 178.50 | bwd_microstep: 302.03 | bwd_inner_microstep: 286.62 | bwd_allreduce_microstep: 15.32 | step_microstep: 21.76 [Rank 0] time (ms) | fwd: 1394.50 | bwd: 2154.74 | bwd_inner: 2138.27 | bwd_allreduce: 15.85 | step: 22.39 4%|▍ | 111/2865 [44:03<15:10:02, 19.83s/it] {'loss': 1.0722, 'learning_rate': 3.9992013182482455e-05, 'epoch': 0.12} 4%|▍ | 111/2865 [44:03<15:10:02, 19.83s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 943 [Rank 0] time (ms) | fwd_microstep: 170.33 | bwd_microstep: 238.09 | bwd_inner_microstep: 237.94 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 167.77 | bwd_microstep: 255.65 | bwd_inner_microstep: 255.48 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 170.36 | bwd_microstep: 268.77 | bwd_inner_microstep: 268.60 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 174.10 | bwd_microstep: 243.18 | bwd_inner_microstep: 242.99 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 167.16 | bwd_microstep: 238.48 | bwd_inner_microstep: 238.35 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 188.75 | bwd_microstep: 251.58 | bwd_inner_microstep: 251.36 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 170.50 | bwd_microstep: 235.55 | bwd_inner_microstep: 235.35 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.01 | optimizer_gradients: 3.03 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 167.27 | bwd_microstep: 300.05 | bwd_inner_microstep: 288.06 | bwd_allreduce_microstep: 11.91 | step_microstep: 22.80 [Rank 0] time (ms) | fwd: 1376.12 | bwd: 2031.48 | bwd_inner: 2018.41 | bwd_allreduce: 12.44 | step: 23.44 4%|▍ | 112/2865 [44:06<11:26:17, 14.96s/it] {'loss': 0.8007, 'learning_rate': 3.99913615050929e-05, 'epoch': 0.12} 4%|▍ | 112/2865 [44:06<11:26:17, 14.96s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 177.13 | bwd_microstep: 290.32 | bwd_inner_microstep: 290.12 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 165.31 | bwd_microstep: 271.06 | bwd_inner_microstep: 270.92 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 181.80 | bwd_microstep: 252.58 | bwd_inner_microstep: 252.35 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 167.71 | bwd_microstep: 245.43 | bwd_inner_microstep: 245.04 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 170.02 | bwd_microstep: 254.63 | bwd_inner_microstep: 254.50 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 175.88 | bwd_microstep: 236.31 | bwd_inner_microstep: 236.14 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 165.49 | bwd_microstep: 271.35 | bwd_inner_microstep: 271.11 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.00 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 171.01 | bwd_microstep: 315.56 | bwd_inner_microstep: 299.60 | bwd_allreduce_microstep: 15.83 | step_microstep: 21.49 [Rank 0] time (ms) | fwd: 1374.25 | bwd: 2137.38 | bwd_inner: 2120.09 | bwd_allreduce: 16.47 | step: 22.23 4%|▍ | 113/2865 [44:10<8:51:07, 11.58s/it] {'loss': 0.9113, 'learning_rate': 3.999068427922857e-05, 'epoch': 0.12} 4%|▍ | 113/2865 [44:10<8:51:07, 11.58s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 167.91 | bwd_microstep: 248.18 | bwd_inner_microstep: 247.99 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 170.66 | bwd_microstep: 249.46 | bwd_inner_microstep: 249.27 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 168.15 | bwd_microstep: 264.95 | bwd_inner_microstep: 264.73 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 171.51 | bwd_microstep: 245.11 | bwd_inner_microstep: 244.91 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 909 [Rank 0] time (ms) | fwd_microstep: 168.26 | bwd_microstep: 256.35 | bwd_inner_microstep: 256.17 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 164.73 | bwd_microstep: 243.39 | bwd_inner_microstep: 243.25 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 153.97 | bwd_microstep: 212.57 | bwd_inner_microstep: 212.27 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 909 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 168.65 | bwd_microstep: 292.67 | bwd_inner_microstep: 282.67 | bwd_allreduce_microstep: 9.91 | step_microstep: 21.06 [Rank 0] time (ms) | fwd: 1333.73 | bwd: 2012.81 | bwd_inner: 2001.52 | bwd_allreduce: 10.58 | step: 21.86 4%|▍ | 114/2865 [44:14<7:00:08, 9.16s/it] {'loss': 1.1036, 'learning_rate': 3.9989981505754946e-05, 'epoch': 0.12} 4%|▍ | 114/2865 [44:14<7:00:08, 9.16s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 169.10 | bwd_microstep: 242.76 | bwd_inner_microstep: 242.52 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 172.52 | bwd_microstep: 242.37 | bwd_inner_microstep: 242.19 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 173.28 | bwd_microstep: 243.59 | bwd_inner_microstep: 243.39 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 167.10 | bwd_microstep: 255.42 | bwd_inner_microstep: 255.24 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 167.73 | bwd_microstep: 240.96 | bwd_inner_microstep: 240.81 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 195.18 | bwd_microstep: 286.95 | bwd_inner_microstep: 286.34 | bwd_allreduce_microstep: 0.35 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 170.48 | bwd_microstep: 257.15 | bwd_inner_microstep: 256.92 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.35 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 176.14 | bwd_microstep: 277.40 | bwd_inner_microstep: 267.21 | bwd_allreduce_microstep: 10.10 | step_microstep: 22.93 [Rank 0] time (ms) | fwd: 1391.41 | bwd: 2046.76 | bwd_inner: 2034.90 | bwd_allreduce: 10.95 | step: 23.88 4%|▍ | 115/2865 [44:17<5:44:07, 7.51s/it] {'loss': 1.2348, 'learning_rate': 3.9989253185570145e-05, 'epoch': 0.12} 4%|▍ | 115/2865 [44:17<5:44:07, 7.51s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 182.19 | bwd_microstep: 255.65 | bwd_inner_microstep: 255.52 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 166.52 | bwd_microstep: 244.64 | bwd_inner_microstep: 244.46 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 175.37 | bwd_microstep: 244.53 | bwd_inner_microstep: 244.37 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.16 | bwd_microstep: 271.86 | bwd_inner_microstep: 271.72 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 170.32 | bwd_microstep: 255.20 | bwd_inner_microstep: 254.99 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 178.00 | bwd_microstep: 246.69 | bwd_inner_microstep: 246.51 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 168.54 | bwd_microstep: 270.80 | bwd_inner_microstep: 270.64 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.97 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 170.95 | bwd_microstep: 315.93 | bwd_inner_microstep: 305.85 | bwd_allreduce_microstep: 9.94 | step_microstep: 21.27 [Rank 0] time (ms) | fwd: 1373.92 | bwd: 2105.41 | bwd_inner: 2094.30 | bwd_allreduce: 10.47 | step: 21.95 4%|▍ | 116/2865 [44:21<4:51:17, 6.36s/it] {'loss': 1.1815, 'learning_rate': 3.9988499319604954e-05, 'epoch': 0.12} 4%|▍ | 116/2865 [44:21<4:51:17, 6.36s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 170.22 | bwd_microstep: 249.55 | bwd_inner_microstep: 249.22 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 205.32 | bwd_microstep: 232.85 | bwd_inner_microstep: 232.67 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 172.78 | bwd_microstep: 256.40 | bwd_inner_microstep: 256.27 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 163.46 | bwd_microstep: 233.52 | bwd_inner_microstep: 233.19 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 165.25 | bwd_microstep: 237.23 | bwd_inner_microstep: 237.12 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 165.88 | bwd_microstep: 243.81 | bwd_inner_microstep: 243.72 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 170.82 | bwd_microstep: 248.93 | bwd_inner_microstep: 248.70 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 909 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.40 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.43 | bwd_microstep: 271.09 | bwd_inner_microstep: 261.04 | bwd_allreduce_microstep: 9.97 | step_microstep: 27.97 [Rank 0] time (ms) | fwd: 1375.02 | bwd: 1973.51 | bwd_inner: 1962.22 | bwd_allreduce: 10.57 | step: 28.69 4%|▍ | 117/2865 [44:25<4:13:52, 5.54s/it] {'loss': 1.0974, 'learning_rate': 3.998771990882279e-05, 'epoch': 0.12} 4%|▍ | 117/2865 [44:25<4:13:52, 5.54s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 174.29 | bwd_microstep: 261.68 | bwd_inner_microstep: 261.51 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 174.67 | bwd_microstep: 232.70 | bwd_inner_microstep: 232.47 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 165.95 | bwd_microstep: 231.23 | bwd_inner_microstep: 231.07 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 169.17 | bwd_microstep: 247.06 | bwd_inner_microstep: 246.93 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 163.88 | bwd_microstep: 239.27 | bwd_inner_microstep: 239.02 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 155.86 | bwd_microstep: 203.91 | bwd_inner_microstep: 203.59 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 163.68 | bwd_microstep: 238.74 | bwd_inner_microstep: 238.55 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 166.81 | bwd_microstep: 329.70 | bwd_inner_microstep: 319.53 | bwd_allreduce_microstep: 10.03 | step_microstep: 21.69 [Rank 0] time (ms) | fwd: 1334.21 | bwd: 1984.41 | bwd_inner: 1972.95 | bwd_allreduce: 10.68 | step: 22.36 4%|▍ | 118/2865 [44:28<3:45:33, 4.93s/it] {'loss': 1.2759, 'learning_rate': 3.998691495421973e-05, 'epoch': 0.12} 4%|▍ | 118/2865 [44:28<3:45:33, 4.93s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 166.82 | bwd_microstep: 264.05 | bwd_inner_microstep: 263.67 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 165.66 | bwd_microstep: 249.88 | bwd_inner_microstep: 249.45 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 170.51 | bwd_microstep: 240.55 | bwd_inner_microstep: 240.34 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 163.04 | bwd_microstep: 235.55 | bwd_inner_microstep: 235.35 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 166.39 | bwd_microstep: 238.30 | bwd_inner_microstep: 238.09 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 169.80 | bwd_microstep: 236.31 | bwd_inner_microstep: 236.18 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 165.09 | bwd_microstep: 232.50 | bwd_inner_microstep: 232.31 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.18 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 164.73 | bwd_microstep: 285.63 | bwd_inner_microstep: 275.17 | bwd_allreduce_microstep: 10.33 | step_microstep: 22.24 [Rank 0] time (ms) | fwd: 1331.92 | bwd: 1982.94 | bwd_inner: 1970.90 | bwd_allreduce: 11.05 | step: 23.00 4%|▍ | 119/2865 [44:32<3:25:55, 4.50s/it] {'loss': 1.1176, 'learning_rate': 3.998608445682447e-05, 'epoch': 0.12} 4%|▍ | 119/2865 [44:32<3:25:55, 4.50s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 938 [Rank 0] time (ms) | fwd_microstep: 168.63 | bwd_microstep: 241.13 | bwd_inner_microstep: 241.00 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 940 [Rank 0] time (ms) | fwd_microstep: 166.84 | bwd_microstep: 244.37 | bwd_inner_microstep: 244.18 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 170.86 | bwd_microstep: 245.05 | bwd_inner_microstep: 244.81 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 168.78 | bwd_microstep: 238.45 | bwd_inner_microstep: 238.25 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 165.79 | bwd_microstep: 256.57 | bwd_inner_microstep: 256.43 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 166.40 | bwd_microstep: 241.63 | bwd_inner_microstep: 241.37 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 166.98 | bwd_microstep: 259.44 | bwd_inner_microstep: 259.29 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.93 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 170.69 | bwd_microstep: 290.55 | bwd_inner_microstep: 279.71 | bwd_allreduce_microstep: 10.74 | step_microstep: 21.76 [Rank 0] time (ms) | fwd: 1344.86 | bwd: 2017.30 | bwd_inner: 2005.29 | bwd_allreduce: 11.33 | step: 22.46 4%|▍ | 120/2865 [44:35<3:12:44, 4.21s/it] {'loss': 1.0881, 'learning_rate': 3.9985228417698375e-05, 'epoch': 0.13} 4%|▍ | 120/2865 [44:35<3:12:44, 4.21s/it][INFO|trainer.py:3242] 2026-05-14 21:05:18,322 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 21:05:18,322 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 21:05:18,322 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-120 [INFO|configuration_utils.py:473] 2026-05-14 21:10:25,230 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-120/config.json [INFO|configuration_utils.py:594] 2026-05-14 21:10:25,240 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-120/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 21:10:30,901 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-120/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 21:10:30,919 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-120/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 21:10:30,929 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-120/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 21:10:30,932 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-120/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 21:10:40,727 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-45] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 303908.87 | bwd_microstep: 229.91 | bwd_inner_microstep: 229.76 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 160.84 | bwd_microstep: 224.44 | bwd_inner_microstep: 224.35 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 161.42 | bwd_microstep: 221.10 | bwd_inner_microstep: 220.98 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 161.10 | bwd_microstep: 218.79 | bwd_inner_microstep: 218.41 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.85 | bwd_microstep: 217.14 | bwd_inner_microstep: 216.77 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 163.63 | bwd_microstep: 241.80 | bwd_inner_microstep: 241.65 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 162.05 | bwd_microstep: 223.20 | bwd_inner_microstep: 222.82 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.95 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 159.85 | bwd_microstep: 250.86 | bwd_inner_microstep: 239.43 | bwd_allreduce_microstep: 11.27 | step_microstep: 21.42 [Rank 0] time (ms) | fwd: 305028.16 | bwd: 1827.39 | bwd_inner: 1814.50 | bwd_allreduce: 12.01 | step: 22.32 4%|▍ | 121/2865 [50:01<76:49:04, 100.78s/it] {'loss': 1.1432, 'learning_rate': 3.998434683793544e-05, 'epoch': 0.13} 4%|▍ | 121/2865 [50:01<76:49:04, 100.78s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 162.13 | bwd_microstep: 217.48 | bwd_inner_microstep: 217.15 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 161.78 | bwd_microstep: 218.90 | bwd_inner_microstep: 218.53 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 164.62 | bwd_microstep: 221.58 | bwd_inner_microstep: 221.45 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.68 | bwd_microstep: 222.12 | bwd_inner_microstep: 221.85 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 162.57 | bwd_microstep: 254.02 | bwd_inner_microstep: 253.83 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.50 | bwd_microstep: 226.86 | bwd_inner_microstep: 226.54 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 162.61 | bwd_microstep: 218.96 | bwd_inner_microstep: 218.64 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 150.26 | bwd_microstep: 231.49 | bwd_inner_microstep: 213.56 | bwd_allreduce_microstep: 17.70 | step_microstep: 21.54 [Rank 0] time (ms) | fwd: 1287.05 | bwd: 1811.53 | bwd_inner: 1792.10 | bwd_allreduce: 18.44 | step: 22.11 4%|▍ | 122/2865 [50:05<54:30:12, 71.53s/it] {'loss': 1.049, 'learning_rate': 3.998343971866231e-05, 'epoch': 0.13} 4%|▍ | 122/2865 [50:05<54:30:12, 71.53s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 164.86 | bwd_microstep: 219.57 | bwd_inner_microstep: 219.30 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 165.28 | bwd_microstep: 220.17 | bwd_inner_microstep: 220.03 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 163.95 | bwd_microstep: 219.92 | bwd_inner_microstep: 219.63 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 163.62 | bwd_microstep: 223.15 | bwd_inner_microstep: 222.81 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 164.39 | bwd_microstep: 220.80 | bwd_inner_microstep: 220.48 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 163.98 | bwd_microstep: 222.23 | bwd_inner_microstep: 221.96 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 162.44 | bwd_microstep: 218.34 | bwd_inner_microstep: 217.99 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 162.80 | bwd_microstep: 253.84 | bwd_inner_microstep: 243.39 | bwd_allreduce_microstep: 10.31 | step_microstep: 21.10 [Rank 0] time (ms) | fwd: 1311.22 | bwd: 1798.12 | bwd_inner: 1786.18 | bwd_allreduce: 10.99 | step: 21.77 4%|▍ | 123/2865 [50:08<38:53:17, 51.06s/it] {'loss': 1.2233, 'learning_rate': 3.9982507061038256e-05, 'epoch': 0.13} 4%|▍ | 123/2865 [50:08<38:53:17, 51.06s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 162.97 | bwd_microstep: 215.67 | bwd_inner_microstep: 215.32 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 148.28 | bwd_microstep: 191.20 | bwd_inner_microstep: 190.87 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 164.30 | bwd_microstep: 252.80 | bwd_inner_microstep: 252.14 | bwd_allreduce_microstep: 0.28 | step_microstep: 0.33 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 166.25 | bwd_microstep: 245.21 | bwd_inner_microstep: 244.84 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.25 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.84 | bwd_microstep: 247.32 | bwd_inner_microstep: 246.91 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 147.82 | bwd_microstep: 188.37 | bwd_inner_microstep: 188.04 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 162.69 | bwd_microstep: 217.02 | bwd_inner_microstep: 216.80 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 162.79 | bwd_microstep: 249.63 | bwd_inner_microstep: 239.32 | bwd_allreduce_microstep: 10.15 | step_microstep: 21.09 [Rank 0] time (ms) | fwd: 1277.81 | bwd: 1807.44 | bwd_inner: 1794.90 | bwd_allreduce: 11.15 | step: 22.25 4%|▍ | 124/2865 [50:11<27:57:26, 36.72s/it] {'loss': 0.9434, 'learning_rate': 3.998154886625519e-05, 'epoch': 0.13} 4%|▍ | 124/2865 [50:11<27:57:26, 36.72s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 163.34 | bwd_microstep: 214.53 | bwd_inner_microstep: 214.20 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 163.30 | bwd_microstep: 219.33 | bwd_inner_microstep: 219.05 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 164.16 | bwd_microstep: 218.45 | bwd_inner_microstep: 217.96 | bwd_allreduce_microstep: 0.22 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 166.80 | bwd_microstep: 219.14 | bwd_inner_microstep: 218.85 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 163.56 | bwd_microstep: 217.06 | bwd_inner_microstep: 216.77 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 163.26 | bwd_microstep: 219.29 | bwd_inner_microstep: 219.00 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.16 | bwd_microstep: 218.95 | bwd_inner_microstep: 218.79 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 175.09 | bwd_microstep: 249.27 | bwd_inner_microstep: 238.26 | bwd_allreduce_microstep: 10.84 | step_microstep: 21.30 [Rank 0] time (ms) | fwd: 1322.54 | bwd: 1776.15 | bwd_inner: 1763.44 | bwd_allreduce: 11.62 | step: 22.02 4%|▍ | 125/2865 [50:14<20:18:43, 26.69s/it] {'loss': 1.1738, 'learning_rate': 3.9980565135537664e-05, 'epoch': 0.13} 4%|▍ | 125/2865 [50:14<20:18:43, 26.69s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 163.93 | bwd_microstep: 225.54 | bwd_inner_microstep: 225.07 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 168.00 | bwd_microstep: 224.49 | bwd_inner_microstep: 223.98 | bwd_allreduce_microstep: 0.23 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 164.09 | bwd_microstep: 229.83 | bwd_inner_microstep: 229.51 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 163.74 | bwd_microstep: 221.22 | bwd_inner_microstep: 220.96 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.39 | bwd_microstep: 222.54 | bwd_inner_microstep: 222.11 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 165.12 | bwd_microstep: 215.89 | bwd_inner_microstep: 215.42 | bwd_allreduce_microstep: 0.21 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 164.52 | bwd_microstep: 220.29 | bwd_inner_microstep: 219.98 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.25 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 164.04 | bwd_microstep: 252.04 | bwd_inner_microstep: 240.51 | bwd_allreduce_microstep: 11.44 | step_microstep: 22.85 [Rank 0] time (ms) | fwd: 1316.70 | bwd: 1812.08 | bwd_inner: 1798.13 | bwd_allreduce: 12.56 | step: 23.72 4%|▍ | 126/2865 [50:18<14:58:12, 19.68s/it] {'loss': 1.1199, 'learning_rate': 3.9979555870142855e-05, 'epoch': 0.13} 4%|▍ | 126/2865 [50:18<14:58:12, 19.68s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 946 [Rank 0] time (ms) | fwd_microstep: 165.52 | bwd_microstep: 222.15 | bwd_inner_microstep: 221.91 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 944 [Rank 0] time (ms) | fwd_microstep: 166.74 | bwd_microstep: 218.43 | bwd_inner_microstep: 218.14 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 166.93 | bwd_microstep: 219.97 | bwd_inner_microstep: 219.77 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 164.21 | bwd_microstep: 191.32 | bwd_inner_microstep: 191.04 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 165.38 | bwd_microstep: 195.55 | bwd_inner_microstep: 195.38 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 165.82 | bwd_microstep: 198.69 | bwd_inner_microstep: 198.23 | bwd_allreduce_microstep: 0.21 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 164.65 | bwd_microstep: 224.19 | bwd_inner_microstep: 223.81 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 164.17 | bwd_microstep: 246.25 | bwd_inner_microstep: 235.69 | bwd_allreduce_microstep: 10.35 | step_microstep: 21.71 [Rank 0] time (ms) | fwd: 1323.29 | bwd: 1716.72 | bwd_inner: 1704.47 | bwd_allreduce: 11.22 | step: 22.56 4%|▍ | 127/2865 [50:21<11:12:37, 14.74s/it] {'loss': 1.3024, 'learning_rate': 3.997852107136058e-05, 'epoch': 0.13} 4%|▍ | 127/2865 [50:21<11:12:37, 14.74s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 941 [Rank 0] time (ms) | fwd_microstep: 164.29 | bwd_microstep: 215.24 | bwd_inner_microstep: 214.85 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 163.99 | bwd_microstep: 218.62 | bwd_inner_microstep: 218.45 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 164.30 | bwd_microstep: 219.69 | bwd_inner_microstep: 219.36 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 164.24 | bwd_microstep: 217.23 | bwd_inner_microstep: 217.09 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 172.86 | bwd_microstep: 218.40 | bwd_inner_microstep: 218.28 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 171.60 | bwd_microstep: 217.58 | bwd_inner_microstep: 217.37 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 173.43 | bwd_microstep: 216.53 | bwd_inner_microstep: 216.40 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.96 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.55 | bwd_microstep: 253.97 | bwd_inner_microstep: 239.86 | bwd_allreduce_microstep: 13.92 | step_microstep: 23.70 [Rank 0] time (ms) | fwd: 1337.13 | bwd: 1777.37 | bwd_inner: 1762.06 | bwd_allreduce: 14.43 | step: 24.29 4%|▍ | 128/2865 [50:24<8:35:52, 11.31s/it] {'loss': 0.9943, 'learning_rate': 3.997746074051331e-05, 'epoch': 0.13} 4%|▍ | 128/2865 [50:24<8:35:52, 11.31s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 163.43 | bwd_microstep: 213.75 | bwd_inner_microstep: 213.47 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 164.03 | bwd_microstep: 220.56 | bwd_inner_microstep: 220.41 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 163.76 | bwd_microstep: 221.44 | bwd_inner_microstep: 221.15 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 163.69 | bwd_microstep: 248.71 | bwd_inner_microstep: 248.47 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 164.19 | bwd_microstep: 217.24 | bwd_inner_microstep: 216.99 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 163.76 | bwd_microstep: 218.68 | bwd_inner_microstep: 218.50 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 166.62 | bwd_microstep: 214.64 | bwd_inner_microstep: 214.26 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.25 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 165.02 | bwd_microstep: 251.70 | bwd_inner_microstep: 241.63 | bwd_allreduce_microstep: 10.00 | step_microstep: 21.18 [Rank 0] time (ms) | fwd: 1314.39 | bwd: 1806.82 | bwd_inner: 1795.35 | bwd_allreduce: 10.59 | step: 21.86 5%|▍ | 129/2865 [50:27<6:46:05, 8.91s/it] {'loss': 1.1173, 'learning_rate': 3.9976374878956096e-05, 'epoch': 0.13} 5%|▍ | 129/2865 [50:27<6:46:05, 8.91s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 163.43 | bwd_microstep: 211.87 | bwd_inner_microstep: 211.62 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 164.92 | bwd_microstep: 215.36 | bwd_inner_microstep: 215.05 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 163.16 | bwd_microstep: 217.65 | bwd_inner_microstep: 217.14 | bwd_allreduce_microstep: 0.21 | step_microstep: 0.25 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 167.89 | bwd_microstep: 220.07 | bwd_inner_microstep: 219.88 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 163.33 | bwd_microstep: 217.13 | bwd_inner_microstep: 216.63 | bwd_allreduce_microstep: 0.22 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.59 | bwd_microstep: 219.77 | bwd_inner_microstep: 219.39 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.26 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.28 | bwd_microstep: 215.54 | bwd_inner_microstep: 215.26 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.95 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 168.65 | bwd_microstep: 246.50 | bwd_inner_microstep: 236.20 | bwd_allreduce_microstep: 10.21 | step_microstep: 21.46 [Rank 0] time (ms) | fwd: 1318.10 | bwd: 1764.15 | bwd_inner: 1751.62 | bwd_allreduce: 11.28 | step: 22.48 5%|▍ | 130/2865 [50:31<5:28:52, 7.21s/it] {'loss': 0.9545, 'learning_rate': 3.997526348807665e-05, 'epoch': 0.14} 5%|▍ | 130/2865 [50:31<5:28:52, 7.21s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 164.72 | bwd_microstep: 211.87 | bwd_inner_microstep: 211.58 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 163.07 | bwd_microstep: 216.45 | bwd_inner_microstep: 216.10 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 164.68 | bwd_microstep: 221.63 | bwd_inner_microstep: 221.46 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 172.70 | bwd_microstep: 219.09 | bwd_inner_microstep: 218.95 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 162.81 | bwd_microstep: 215.25 | bwd_inner_microstep: 214.87 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.26 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 163.54 | bwd_microstep: 213.53 | bwd_inner_microstep: 213.32 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 168.23 | bwd_microstep: 217.72 | bwd_inner_microstep: 217.49 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.01 | optimizer_gradients: 2.94 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 162.78 | bwd_microstep: 257.42 | bwd_inner_microstep: 246.44 | bwd_allreduce_microstep: 10.81 | step_microstep: 21.44 [Rank 0] time (ms) | fwd: 1322.40 | bwd: 1773.08 | bwd_inner: 1760.75 | bwd_allreduce: 11.37 | step: 22.22 5%|▍ | 131/2865 [50:34<4:34:51, 6.03s/it] {'loss': 1.0374, 'learning_rate': 3.99741265692953e-05, 'epoch': 0.14} 5%|▍ | 131/2865 [50:34<4:34:51, 6.03s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 172.59 | bwd_microstep: 219.48 | bwd_inner_microstep: 219.17 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 163.34 | bwd_microstep: 219.09 | bwd_inner_microstep: 218.88 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 162.87 | bwd_microstep: 213.87 | bwd_inner_microstep: 213.71 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 162.57 | bwd_microstep: 216.33 | bwd_inner_microstep: 216.10 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 172.51 | bwd_microstep: 214.50 | bwd_inner_microstep: 214.28 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 163.01 | bwd_microstep: 214.91 | bwd_inner_microstep: 214.79 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 163.49 | bwd_microstep: 215.76 | bwd_inner_microstep: 215.34 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.97 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 163.49 | bwd_microstep: 267.10 | bwd_inner_microstep: 253.63 | bwd_allreduce_microstep: 13.37 | step_microstep: 23.48 [Rank 0] time (ms) | fwd: 1323.75 | bwd: 1781.20 | bwd_inner: 1766.25 | bwd_allreduce: 14.05 | step: 24.16 5%|▍ | 132/2865 [50:37<3:57:08, 5.21s/it] {'loss': 0.908, 'learning_rate': 3.997296412406502e-05, 'epoch': 0.14} 5%|▍ | 132/2865 [50:37<3:57:08, 5.21s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 943 [Rank 0] time (ms) | fwd_microstep: 165.68 | bwd_microstep: 217.57 | bwd_inner_microstep: 217.32 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 164.32 | bwd_microstep: 223.08 | bwd_inner_microstep: 222.67 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 164.50 | bwd_microstep: 217.85 | bwd_inner_microstep: 217.54 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 163.57 | bwd_microstep: 227.55 | bwd_inner_microstep: 227.39 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 162.59 | bwd_microstep: 221.18 | bwd_inner_microstep: 220.87 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 166.18 | bwd_microstep: 223.50 | bwd_inner_microstep: 223.18 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 172.47 | bwd_microstep: 217.25 | bwd_inner_microstep: 216.95 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 174.92 | bwd_microstep: 248.44 | bwd_inner_microstep: 237.84 | bwd_allreduce_microstep: 10.49 | step_microstep: 21.27 [Rank 0] time (ms) | fwd: 1334.08 | bwd: 1796.54 | bwd_inner: 1784.36 | bwd_allreduce: 11.13 | step: 21.94 5%|▍ | 133/2865 [50:41<3:31:10, 4.64s/it] {'loss': 1.1433, 'learning_rate': 3.997177615387136e-05, 'epoch': 0.14} 5%|▍ | 133/2865 [50:41<3:31:10, 4.64s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 171.92 | bwd_microstep: 216.61 | bwd_inner_microstep: 216.28 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 175.90 | bwd_microstep: 216.76 | bwd_inner_microstep: 216.39 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 169.66 | bwd_microstep: 217.85 | bwd_inner_microstep: 217.49 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.25 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 163.15 | bwd_microstep: 254.49 | bwd_inner_microstep: 254.20 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 162.57 | bwd_microstep: 219.82 | bwd_inner_microstep: 219.51 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 163.99 | bwd_microstep: 223.13 | bwd_inner_microstep: 222.95 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 169.57 | bwd_microstep: 220.11 | bwd_inner_microstep: 219.98 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 165.35 | bwd_microstep: 251.86 | bwd_inner_microstep: 241.55 | bwd_allreduce_microstep: 10.21 | step_microstep: 21.25 [Rank 0] time (ms) | fwd: 1341.97 | bwd: 1820.77 | bwd_inner: 1808.81 | bwd_allreduce: 10.95 | step: 22.05 5%|▍ | 134/2865 [50:44<3:13:24, 4.25s/it] {'loss': 1.0295, 'learning_rate': 3.9970562660232525e-05, 'epoch': 0.14} 5%|▍ | 134/2865 [50:44<3:13:24, 4.25s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 163.39 | bwd_microstep: 218.49 | bwd_inner_microstep: 218.14 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 163.32 | bwd_microstep: 218.56 | bwd_inner_microstep: 218.18 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.24 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 163.07 | bwd_microstep: 215.80 | bwd_inner_microstep: 215.67 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 164.34 | bwd_microstep: 218.06 | bwd_inner_microstep: 217.77 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 163.77 | bwd_microstep: 216.08 | bwd_inner_microstep: 215.93 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 163.76 | bwd_microstep: 217.62 | bwd_inner_microstep: 217.43 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 165.08 | bwd_microstep: 215.61 | bwd_inner_microstep: 215.21 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.41 | bwd_microstep: 250.75 | bwd_inner_microstep: 240.41 | bwd_allreduce_microstep: 10.26 | step_microstep: 21.38 [Rank 0] time (ms) | fwd: 1310.04 | bwd: 1771.13 | bwd_inner: 1759.13 | bwd_allreduce: 11.02 | step: 22.24 5%|▍ | 135/2865 [50:47<2:59:38, 3.95s/it] {'loss': 1.0962, 'learning_rate': 3.996932364469934e-05, 'epoch': 0.14} 5%|▍ | 135/2865 [50:47<2:59:38, 3.95s/it][INFO|trainer.py:3242] 2026-05-14 21:11:30,398 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 21:11:30,398 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 21:11:30,398 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-135 [INFO|configuration_utils.py:473] 2026-05-14 21:16:35,496 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-135/config.json [INFO|configuration_utils.py:594] 2026-05-14 21:16:35,511 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-135/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 21:16:40,594 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-135/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 21:16:40,615 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-135/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 21:16:40,621 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-135/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 21:16:40,623 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-135/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 21:16:51,569 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-60] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 302330.42 | bwd_microstep: 217.87 | bwd_inner_microstep: 217.72 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 159.63 | bwd_microstep: 216.95 | bwd_inner_microstep: 216.67 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 160.25 | bwd_microstep: 217.83 | bwd_inner_microstep: 217.70 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 159.84 | bwd_microstep: 217.37 | bwd_inner_microstep: 217.25 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 160.05 | bwd_microstep: 219.45 | bwd_inner_microstep: 219.09 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 160.16 | bwd_microstep: 216.24 | bwd_inner_microstep: 215.95 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.30 | bwd_microstep: 213.32 | bwd_inner_microstep: 212.95 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.66 | bwd_microstep: 246.52 | bwd_inner_microstep: 236.69 | bwd_allreduce_microstep: 9.76 | step_microstep: 21.21 [Rank 0] time (ms) | fwd: 303441.96 | bwd: 1765.68 | bwd_inner: 1754.36 | bwd_allreduce: 10.49 | step: 21.94 5%|▍ | 136/2865 [56:12<75:56:55, 100.19s/it] {'loss': 0.9025, 'learning_rate': 3.996805910885523e-05, 'epoch': 0.14} 5%|▍ | 136/2865 [56:12<75:56:55, 100.19s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 160.85 | bwd_microstep: 212.66 | bwd_inner_microstep: 212.52 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 160.87 | bwd_microstep: 216.71 | bwd_inner_microstep: 216.29 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.96 | bwd_microstep: 222.50 | bwd_inner_microstep: 222.29 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.92 | bwd_microstep: 219.82 | bwd_inner_microstep: 219.68 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.46 | bwd_microstep: 219.71 | bwd_inner_microstep: 219.38 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 159.84 | bwd_microstep: 218.70 | bwd_inner_microstep: 218.33 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 160.35 | bwd_microstep: 216.07 | bwd_inner_microstep: 215.91 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 159.93 | bwd_microstep: 251.04 | bwd_inner_microstep: 241.30 | bwd_allreduce_microstep: 9.66 | step_microstep: 21.21 [Rank 0] time (ms) | fwd: 1285.07 | bwd: 1777.38 | bwd_inner: 1766.11 | bwd_allreduce: 10.34 | step: 22.04 5%|▍ | 137/2865 [56:15<53:52:45, 71.10s/it] {'loss': 0.804, 'learning_rate': 3.996676905431623e-05, 'epoch': 0.14} 5%|▍ | 137/2865 [56:15<53:52:45, 71.10s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 160.67 | bwd_microstep: 214.24 | bwd_inner_microstep: 213.95 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 159.85 | bwd_microstep: 217.89 | bwd_inner_microstep: 217.76 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 159.30 | bwd_microstep: 218.72 | bwd_inner_microstep: 218.49 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 159.29 | bwd_microstep: 214.27 | bwd_inner_microstep: 213.88 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 159.14 | bwd_microstep: 217.68 | bwd_inner_microstep: 217.55 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 146.01 | bwd_microstep: 190.01 | bwd_inner_microstep: 189.86 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 159.79 | bwd_microstep: 219.43 | bwd_inner_microstep: 219.03 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 159.75 | bwd_microstep: 247.64 | bwd_inner_microstep: 236.61 | bwd_allreduce_microstep: 10.84 | step_microstep: 21.07 [Rank 0] time (ms) | fwd: 1263.69 | bwd: 1740.02 | bwd_inner: 1727.60 | bwd_allreduce: 11.51 | step: 21.80 5%|▍ | 138/2865 [56:18<38:25:13, 50.72s/it] {'loss': 1.1246, 'learning_rate': 3.996545348273102e-05, 'epoch': 0.14} 5%|▍ | 138/2865 [56:18<38:25:13, 50.72s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 146.61 | bwd_microstep: 188.05 | bwd_inner_microstep: 187.75 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.13 | bwd_microstep: 222.68 | bwd_inner_microstep: 222.28 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.17 | bwd_microstep: 220.35 | bwd_inner_microstep: 220.03 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 160.63 | bwd_microstep: 218.48 | bwd_inner_microstep: 218.35 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 159.90 | bwd_microstep: 217.63 | bwd_inner_microstep: 217.49 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.95 | bwd_microstep: 220.66 | bwd_inner_microstep: 220.26 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.88 | bwd_microstep: 215.90 | bwd_inner_microstep: 215.77 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 159.88 | bwd_microstep: 222.25 | bwd_inner_microstep: 212.42 | bwd_allreduce_microstep: 9.67 | step_microstep: 21.22 [Rank 0] time (ms) | fwd: 1271.05 | bwd: 1726.12 | bwd_inner: 1714.77 | bwd_allreduce: 10.37 | step: 21.95 5%|▍ | 139/2865 [56:22<27:36:02, 36.45s/it] {'loss': 0.9042, 'learning_rate': 3.9964112395780866e-05, 'epoch': 0.15} 5%|▍ | 139/2865 [56:22<27:36:02, 36.45s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 162.33 | bwd_microstep: 193.21 | bwd_inner_microstep: 193.07 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.77 | bwd_microstep: 190.85 | bwd_inner_microstep: 190.54 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.79 | bwd_microstep: 192.50 | bwd_inner_microstep: 192.13 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 161.02 | bwd_microstep: 193.82 | bwd_inner_microstep: 193.43 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.24 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 162.43 | bwd_microstep: 189.41 | bwd_inner_microstep: 189.02 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 160.09 | bwd_microstep: 193.88 | bwd_inner_microstep: 193.78 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 160.40 | bwd_microstep: 192.41 | bwd_inner_microstep: 192.06 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 147.05 | bwd_microstep: 198.67 | bwd_inner_microstep: 188.73 | bwd_allreduce_microstep: 9.84 | step_microstep: 21.31 [Rank 0] time (ms) | fwd: 1275.80 | bwd: 1544.92 | bwd_inner: 1533.16 | bwd_allreduce: 10.71 | step: 22.15 5%|▍ | 140/2865 [56:24<19:59:22, 26.41s/it] {'loss': 1.259, 'learning_rate': 3.996274579517962e-05, 'epoch': 0.15} 5%|▍ | 140/2865 [56:24<19:59:22, 26.41s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.09 | bwd_microstep: 192.01 | bwd_inner_microstep: 191.69 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 163.49 | bwd_microstep: 196.22 | bwd_inner_microstep: 196.03 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 160.48 | bwd_microstep: 194.92 | bwd_inner_microstep: 194.75 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.31 | bwd_microstep: 191.65 | bwd_inner_microstep: 191.36 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 163.93 | bwd_microstep: 194.04 | bwd_inner_microstep: 193.58 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 163.44 | bwd_microstep: 189.57 | bwd_inner_microstep: 189.26 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.47 | bwd_microstep: 188.92 | bwd_inner_microstep: 188.81 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.97 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.14 | bwd_microstep: 229.17 | bwd_inner_microstep: 216.80 | bwd_allreduce_microstep: 12.21 | step_microstep: 21.56 [Rank 0] time (ms) | fwd: 1298.23 | bwd: 1576.64 | bwd_inner: 1562.79 | bwd_allreduce: 12.91 | step: 22.19 5%|▍ | 141/2865 [56:28<14:40:50, 19.40s/it] {'loss': 0.8714, 'learning_rate': 3.996135368267379e-05, 'epoch': 0.15} 5%|▍ | 141/2865 [56:28<14:40:50, 19.40s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 164.55 | bwd_microstep: 191.10 | bwd_inner_microstep: 190.94 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 941 [Rank 0] time (ms) | fwd_microstep: 164.33 | bwd_microstep: 193.54 | bwd_inner_microstep: 193.20 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 163.02 | bwd_microstep: 187.80 | bwd_inner_microstep: 187.30 | bwd_allreduce_microstep: 0.21 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 163.55 | bwd_microstep: 217.53 | bwd_inner_microstep: 217.35 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 175.64 | bwd_microstep: 215.69 | bwd_inner_microstep: 215.53 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 166.59 | bwd_microstep: 219.12 | bwd_inner_microstep: 218.94 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 166.93 | bwd_microstep: 221.79 | bwd_inner_microstep: 221.67 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.91 | bwd_microstep: 249.21 | bwd_inner_microstep: 239.05 | bwd_allreduce_microstep: 10.09 | step_microstep: 21.01 [Rank 0] time (ms) | fwd: 1327.38 | bwd: 1695.91 | bwd_inner: 1684.31 | bwd_allreduce: 10.73 | step: 21.68 5%|▍ | 142/2865 [56:31<10:59:58, 14.54s/it] {'loss': 1.0765, 'learning_rate': 3.995993606004246e-05, 'epoch': 0.15} 5%|▍ | 142/2865 [56:31<10:59:58, 14.54s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 163.64 | bwd_microstep: 213.80 | bwd_inner_microstep: 213.51 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 163.27 | bwd_microstep: 216.93 | bwd_inner_microstep: 216.68 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 163.24 | bwd_microstep: 218.60 | bwd_inner_microstep: 218.47 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 162.31 | bwd_microstep: 188.69 | bwd_inner_microstep: 188.39 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 162.75 | bwd_microstep: 192.29 | bwd_inner_microstep: 192.16 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 164.62 | bwd_microstep: 191.21 | bwd_inner_microstep: 191.07 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 165.30 | bwd_microstep: 193.02 | bwd_inner_microstep: 192.85 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 165.91 | bwd_microstep: 229.50 | bwd_inner_microstep: 218.95 | bwd_allreduce_microstep: 10.45 | step_microstep: 21.76 [Rank 0] time (ms) | fwd: 1310.92 | bwd: 1644.13 | bwd_inner: 1632.43 | bwd_allreduce: 10.96 | step: 22.33 5%|▍ | 143/2865 [56:34<8:24:23, 11.12s/it] {'loss': 1.0438, 'learning_rate': 3.9958492929097306e-05, 'epoch': 0.15} 5%|▍ | 143/2865 [56:34<8:24:23, 11.12s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 164.39 | bwd_microstep: 188.91 | bwd_inner_microstep: 188.57 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 164.08 | bwd_microstep: 195.33 | bwd_inner_microstep: 195.17 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 164.14 | bwd_microstep: 190.06 | bwd_inner_microstep: 189.93 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.97 | bwd_microstep: 190.73 | bwd_inner_microstep: 190.60 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 160.83 | bwd_microstep: 194.11 | bwd_inner_microstep: 193.69 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 161.36 | bwd_microstep: 192.26 | bwd_inner_microstep: 192.01 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 160.83 | bwd_microstep: 192.80 | bwd_inner_microstep: 192.33 | bwd_allreduce_microstep: 0.21 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 160.49 | bwd_microstep: 223.42 | bwd_inner_microstep: 210.82 | bwd_allreduce_microstep: 12.44 | step_microstep: 21.13 [Rank 0] time (ms) | fwd: 1297.99 | bwd: 1567.77 | bwd_inner: 1553.57 | bwd_allreduce: 13.15 | step: 21.79 5%|▌ | 144/2865 [56:37<6:34:09, 8.69s/it] {'loss': 1.0476, 'learning_rate': 3.9957024291682615e-05, 'epoch': 0.15} 5%|▌ | 144/2865 [56:37<6:34:09, 8.69s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 155.29 | bwd_microstep: 211.78 | bwd_inner_microstep: 211.47 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 156.78 | bwd_microstep: 218.28 | bwd_inner_microstep: 218.10 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 156.00 | bwd_microstep: 217.65 | bwd_inner_microstep: 217.32 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 156.66 | bwd_microstep: 216.05 | bwd_inner_microstep: 215.66 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 155.53 | bwd_microstep: 240.59 | bwd_inner_microstep: 240.48 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 155.12 | bwd_microstep: 215.40 | bwd_inner_microstep: 215.02 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 155.74 | bwd_microstep: 220.97 | bwd_inner_microstep: 220.60 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 156.67 | bwd_microstep: 253.08 | bwd_inner_microstep: 244.71 | bwd_allreduce_microstep: 8.20 | step_microstep: 21.15 [Rank 0] time (ms) | fwd: 1247.70 | bwd: 1793.95 | bwd_inner: 1783.88 | bwd_allreduce: 8.97 | step: 21.99 5%|▌ | 145/2865 [56:40<5:19:12, 7.04s/it] {'loss': 0.9741, 'learning_rate': 3.995553014967528e-05, 'epoch': 0.15} 5%|▌ | 145/2865 [56:40<5:19:12, 7.04s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 156.30 | bwd_microstep: 224.34 | bwd_inner_microstep: 224.03 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 155.50 | bwd_microstep: 224.02 | bwd_inner_microstep: 223.64 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 156.09 | bwd_microstep: 218.05 | bwd_inner_microstep: 217.67 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.17 | bwd_microstep: 216.10 | bwd_inner_microstep: 215.95 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 155.89 | bwd_microstep: 218.00 | bwd_inner_microstep: 217.69 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.08 | bwd_microstep: 218.45 | bwd_inner_microstep: 218.14 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 165.64 | bwd_microstep: 219.42 | bwd_inner_microstep: 219.20 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.94 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 149.62 | bwd_microstep: 221.95 | bwd_inner_microstep: 211.54 | bwd_allreduce_microstep: 10.27 | step_microstep: 21.68 [Rank 0] time (ms) | fwd: 1262.19 | bwd: 1760.47 | bwd_inner: 1748.34 | bwd_allreduce: 11.11 | step: 22.45 5%|▌ | 146/2865 [56:43<4:26:31, 5.88s/it] {'loss': 1.0862, 'learning_rate': 3.995401050498477e-05, 'epoch': 0.15} 5%|▌ | 146/2865 [56:43<4:26:31, 5.88s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 166.37 | bwd_microstep: 215.22 | bwd_inner_microstep: 214.86 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 164.05 | bwd_microstep: 217.71 | bwd_inner_microstep: 217.53 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 164.19 | bwd_microstep: 221.07 | bwd_inner_microstep: 220.76 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 164.65 | bwd_microstep: 224.85 | bwd_inner_microstep: 224.70 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 165.20 | bwd_microstep: 223.68 | bwd_inner_microstep: 223.45 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 163.92 | bwd_microstep: 219.74 | bwd_inner_microstep: 219.59 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 169.25 | bwd_microstep: 222.21 | bwd_inner_microstep: 221.83 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.93 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.15 | bwd_microstep: 254.78 | bwd_inner_microstep: 242.89 | bwd_allreduce_microstep: 11.69 | step_microstep: 21.42 [Rank 0] time (ms) | fwd: 1320.67 | bwd: 1799.39 | bwd_inner: 1786.05 | bwd_allreduce: 12.30 | step: 22.15 5%|▌ | 147/2865 [56:47<3:51:29, 5.11s/it] {'loss': 1.1516, 'learning_rate': 3.995246535955315e-05, 'epoch': 0.15} 5%|▌ | 147/2865 [56:47<3:51:29, 5.11s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 163.08 | bwd_microstep: 217.82 | bwd_inner_microstep: 217.50 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 164.09 | bwd_microstep: 217.20 | bwd_inner_microstep: 216.91 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 165.75 | bwd_microstep: 216.28 | bwd_inner_microstep: 216.00 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 167.07 | bwd_microstep: 216.62 | bwd_inner_microstep: 216.35 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 165.74 | bwd_microstep: 216.45 | bwd_inner_microstep: 216.09 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 179.39 | bwd_microstep: 215.05 | bwd_inner_microstep: 214.88 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 167.74 | bwd_microstep: 216.42 | bwd_inner_microstep: 216.11 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 168.87 | bwd_microstep: 251.42 | bwd_inner_microstep: 238.45 | bwd_allreduce_microstep: 12.81 | step_microstep: 22.92 [Rank 0] time (ms) | fwd: 1341.60 | bwd: 1767.40 | bwd_inner: 1752.77 | bwd_allreduce: 13.64 | step: 23.63 5%|▌ | 148/2865 [56:50<3:26:54, 4.57s/it] {'loss': 0.994, 'learning_rate': 3.99508947153551e-05, 'epoch': 0.15} 5%|▌ | 148/2865 [56:50<3:26:54, 4.57s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 941 [Rank 0] time (ms) | fwd_microstep: 173.95 | bwd_microstep: 214.95 | bwd_inner_microstep: 214.67 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 169.58 | bwd_microstep: 218.23 | bwd_inner_microstep: 217.99 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 162.29 | bwd_microstep: 224.09 | bwd_inner_microstep: 223.77 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.89 | bwd_microstep: 238.17 | bwd_inner_microstep: 237.79 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 162.98 | bwd_microstep: 220.96 | bwd_inner_microstep: 220.82 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.20 | bwd_microstep: 221.45 | bwd_inner_microstep: 221.07 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 160.28 | bwd_microstep: 222.22 | bwd_inner_microstep: 222.00 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.67 | bwd_microstep: 252.83 | bwd_inner_microstep: 241.17 | bwd_allreduce_microstep: 11.50 | step_microstep: 21.03 [Rank 0] time (ms) | fwd: 1315.70 | bwd: 1813.04 | bwd_inner: 1799.81 | bwd_allreduce: 12.21 | step: 21.79 5%|▌ | 149/2865 [56:53<3:09:37, 4.19s/it] {'loss': 1.059, 'learning_rate': 3.994929857439784e-05, 'epoch': 0.16} 5%|▌ | 149/2865 [56:53<3:09:37, 4.19s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 161.67 | bwd_microstep: 215.18 | bwd_inner_microstep: 214.99 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.34 | bwd_microstep: 217.49 | bwd_inner_microstep: 217.35 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 160.36 | bwd_microstep: 232.62 | bwd_inner_microstep: 232.48 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 161.98 | bwd_microstep: 231.63 | bwd_inner_microstep: 231.50 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.59 | bwd_microstep: 248.79 | bwd_inner_microstep: 248.52 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 160.63 | bwd_microstep: 220.77 | bwd_inner_microstep: 220.60 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.13 | bwd_microstep: 224.12 | bwd_inner_microstep: 223.70 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.43 | bwd_microstep: 256.54 | bwd_inner_microstep: 246.29 | bwd_allreduce_microstep: 10.08 | step_microstep: 21.21 [Rank 0] time (ms) | fwd: 1288.02 | bwd: 1847.26 | bwd_inner: 1835.80 | bwd_allreduce: 10.63 | step: 21.80 5%|▌ | 150/2865 [56:56<2:57:25, 3.92s/it] {'loss': 0.9952, 'learning_rate': 3.9947676938721206e-05, 'epoch': 0.16} 5%|▌ | 150/2865 [56:56<2:57:25, 3.92s/it][INFO|trainer.py:3242] 2026-05-14 21:17:39,668 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 21:17:39,668 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 21:17:39,669 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-150 [INFO|configuration_utils.py:473] 2026-05-14 21:22:45,086 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-150/config.json [INFO|configuration_utils.py:594] 2026-05-14 21:22:45,098 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-150/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 21:22:51,402 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-150/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 21:22:51,418 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-150/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 21:22:51,428 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-150/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 21:22:51,432 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-150/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 21:23:01,033 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-75] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 302423.60 | bwd_microstep: 295.48 | bwd_inner_microstep: 295.20 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 173.51 | bwd_microstep: 300.82 | bwd_inner_microstep: 300.66 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 167.41 | bwd_microstep: 260.90 | bwd_inner_microstep: 260.52 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 165.48 | bwd_microstep: 294.70 | bwd_inner_microstep: 294.52 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 176.39 | bwd_microstep: 261.83 | bwd_inner_microstep: 261.66 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 173.56 | bwd_microstep: 263.84 | bwd_inner_microstep: 263.68 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 155.88 | bwd_microstep: 228.66 | bwd_inner_microstep: 228.26 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.05 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 180.72 | bwd_microstep: 302.31 | bwd_inner_microstep: 290.54 | bwd_allreduce_microstep: 11.61 | step_microstep: 22.07 [Rank 0] time (ms) | fwd: 303607.03 | bwd: 2208.67 | bwd_inner: 2195.42 | bwd_allreduce: 12.33 | step: 22.93 5%|▌ | 151/2865 [1:02:22<75:43:52, 100.45s/it] {'loss': 1.1146, 'learning_rate': 3.994602981039761e-05, 'epoch': 0.16} 5%|▌ | 151/2865 [1:02:22<75:43:52, 100.45s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 174.88 | bwd_microstep: 256.57 | bwd_inner_microstep: 256.30 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 175.67 | bwd_microstep: 261.66 | bwd_inner_microstep: 261.34 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 182.05 | bwd_microstep: 309.25 | bwd_inner_microstep: 308.81 | bwd_allreduce_microstep: 0.20 | step_microstep: 0.24 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 177.69 | bwd_microstep: 304.32 | bwd_inner_microstep: 304.15 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 165.65 | bwd_microstep: 244.14 | bwd_inner_microstep: 243.82 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 163.42 | bwd_microstep: 231.57 | bwd_inner_microstep: 231.24 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 182.02 | bwd_microstep: 262.21 | bwd_inner_microstep: 261.96 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.04 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 178.28 | bwd_microstep: 304.88 | bwd_inner_microstep: 293.69 | bwd_allreduce_microstep: 10.96 | step_microstep: 23.51 [Rank 0] time (ms) | fwd: 1399.55 | bwd: 2174.72 | bwd_inner: 2161.78 | bwd_allreduce: 11.77 | step: 24.39 5%|▌ | 152/2865 [1:02:26<53:50:55, 71.45s/it] {'loss': 1.0162, 'learning_rate': 3.994435719153206e-05, 'epoch': 0.16} 5%|▌ | 152/2865 [1:02:26<53:50:55, 71.45s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 184.78 | bwd_microstep: 291.01 | bwd_inner_microstep: 290.73 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 163.70 | bwd_microstep: 225.95 | bwd_inner_microstep: 225.66 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 171.06 | bwd_microstep: 225.11 | bwd_inner_microstep: 224.78 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 171.51 | bwd_microstep: 223.59 | bwd_inner_microstep: 223.17 | bwd_allreduce_microstep: 0.21 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 167.75 | bwd_microstep: 218.53 | bwd_inner_microstep: 218.40 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.25 | bwd_microstep: 221.31 | bwd_inner_microstep: 221.18 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.46 | bwd_microstep: 217.13 | bwd_inner_microstep: 216.82 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.34 | bwd_microstep: 252.94 | bwd_inner_microstep: 241.99 | bwd_allreduce_microstep: 10.78 | step_microstep: 21.29 [Rank 0] time (ms) | fwd: 1339.69 | bwd: 1875.68 | bwd_inner: 1863.18 | bwd_allreduce: 11.54 | step: 21.98 5%|▌ | 153/2865 [1:02:29<38:26:58, 51.04s/it] {'loss': 1.0621, 'learning_rate': 3.9942659084262086e-05, 'epoch': 0.16} 5%|▌ | 153/2865 [1:02:29<38:26:58, 51.04s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.60 | bwd_microstep: 217.92 | bwd_inner_microstep: 217.70 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 161.74 | bwd_microstep: 218.90 | bwd_inner_microstep: 218.50 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.28 | bwd_microstep: 216.61 | bwd_inner_microstep: 216.21 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.43 | bwd_microstep: 215.43 | bwd_inner_microstep: 215.30 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 160.65 | bwd_microstep: 217.58 | bwd_inner_microstep: 217.44 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 160.97 | bwd_microstep: 213.57 | bwd_inner_microstep: 213.21 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.22 | bwd_microstep: 227.47 | bwd_inner_microstep: 227.34 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.69 | bwd_microstep: 246.10 | bwd_inner_microstep: 235.73 | bwd_allreduce_microstep: 10.21 | step_microstep: 21.21 [Rank 0] time (ms) | fwd: 1286.48 | bwd: 1773.74 | bwd_inner: 1761.94 | bwd_allreduce: 10.91 | step: 22.07 5%|▌ | 154/2865 [1:02:33<27:38:01, 36.70s/it] {'loss': 0.8203, 'learning_rate': 3.9940935490757864e-05, 'epoch': 0.16} 5%|▌ | 154/2865 [1:02:33<27:38:01, 36.70s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 164.70 | bwd_microstep: 218.25 | bwd_inner_microstep: 218.06 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.55 | bwd_microstep: 227.14 | bwd_inner_microstep: 227.01 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 162.33 | bwd_microstep: 213.60 | bwd_inner_microstep: 213.46 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 170.09 | bwd_microstep: 244.58 | bwd_inner_microstep: 244.22 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 162.95 | bwd_microstep: 222.10 | bwd_inner_microstep: 221.98 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 163.52 | bwd_microstep: 219.69 | bwd_inner_microstep: 219.39 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 163.72 | bwd_microstep: 262.77 | bwd_inner_microstep: 262.47 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.86 [Rank 0] time (ms) | fwd_microstep: 161.61 | bwd_microstep: 282.69 | bwd_inner_microstep: 271.97 | bwd_allreduce_microstep: 10.56 | step_microstep: 21.20 [Rank 0] time (ms) | fwd: 1310.35 | bwd: 1890.94 | bwd_inner: 1878.93 | bwd_allreduce: 11.19 | step: 21.88 5%|▌ | 155/2865 [1:02:36<20:05:58, 26.70s/it] {'loss': 0.9784, 'learning_rate': 3.993918641322208e-05, 'epoch': 0.16} 5%|▌ | 155/2865 [1:02:36<20:05:58, 26.70s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 160.88 | bwd_microstep: 222.09 | bwd_inner_microstep: 221.82 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 172.31 | bwd_microstep: 221.46 | bwd_inner_microstep: 221.33 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.68 | bwd_microstep: 218.60 | bwd_inner_microstep: 218.24 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 164.65 | bwd_microstep: 221.82 | bwd_inner_microstep: 221.51 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.93 | bwd_microstep: 226.51 | bwd_inner_microstep: 226.19 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.21 | bwd_microstep: 220.24 | bwd_inner_microstep: 220.11 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 161.04 | bwd_microstep: 221.44 | bwd_inner_microstep: 221.03 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 164.88 | bwd_microstep: 251.30 | bwd_inner_microstep: 240.39 | bwd_allreduce_microstep: 10.76 | step_microstep: 21.31 [Rank 0] time (ms) | fwd: 1307.44 | bwd: 1803.60 | bwd_inner: 1791.06 | bwd_allreduce: 11.50 | step: 22.05 5%|▌ | 156/2865 [1:02:39<14:48:37, 19.68s/it] {'loss': 1.2086, 'learning_rate': 3.993741185389003e-05, 'epoch': 0.16} 5%|▌ | 156/2865 [1:02:39<14:48:37, 19.68s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 165.99 | bwd_microstep: 216.58 | bwd_inner_microstep: 216.20 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.44 | bwd_microstep: 219.42 | bwd_inner_microstep: 219.14 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 164.27 | bwd_microstep: 224.00 | bwd_inner_microstep: 223.67 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 162.75 | bwd_microstep: 229.50 | bwd_inner_microstep: 229.13 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 163.98 | bwd_microstep: 222.85 | bwd_inner_microstep: 222.45 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.25 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 163.26 | bwd_microstep: 226.52 | bwd_inner_microstep: 226.14 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 161.11 | bwd_microstep: 220.01 | bwd_inner_microstep: 219.83 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.37 | bwd_microstep: 255.76 | bwd_inner_microstep: 243.84 | bwd_allreduce_microstep: 11.77 | step_microstep: 21.51 [Rank 0] time (ms) | fwd: 1304.06 | bwd: 1814.83 | bwd_inner: 1800.93 | bwd_allreduce: 12.67 | step: 22.61 5%|▌ | 157/2865 [1:02:43<11:06:30, 14.77s/it] {'loss': 0.9117, 'learning_rate': 3.993561181502954e-05, 'epoch': 0.16} 5%|▌ | 157/2865 [1:02:43<11:06:30, 14.77s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.85 | bwd_microstep: 218.97 | bwd_inner_microstep: 218.67 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.99 | bwd_microstep: 235.39 | bwd_inner_microstep: 235.26 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 165.49 | bwd_microstep: 215.89 | bwd_inner_microstep: 215.77 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.01 | bwd_microstep: 215.09 | bwd_inner_microstep: 214.77 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.16 | bwd_microstep: 215.67 | bwd_inner_microstep: 215.53 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.07 | bwd_microstep: 213.95 | bwd_inner_microstep: 213.74 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 163.00 | bwd_microstep: 215.19 | bwd_inner_microstep: 215.01 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 160.72 | bwd_microstep: 248.08 | bwd_inner_microstep: 237.25 | bwd_allreduce_microstep: 10.66 | step_microstep: 20.97 [Rank 0] time (ms) | fwd: 1296.19 | bwd: 1778.29 | bwd_inner: 1766.42 | bwd_allreduce: 11.11 | step: 21.46 6%|▌ | 158/2865 [1:02:46<8:30:10, 11.31s/it] {'loss': 1.0186, 'learning_rate': 3.9933786298941036e-05, 'epoch': 0.17} 6%|▌ | 158/2865 [1:02:46<8:30:10, 11.31s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 161.62 | bwd_microstep: 214.81 | bwd_inner_microstep: 214.37 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 216.50 | bwd_microstep: 215.68 | bwd_inner_microstep: 215.31 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.24 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 161.72 | bwd_microstep: 217.97 | bwd_inner_microstep: 217.75 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 161.50 | bwd_microstep: 212.83 | bwd_inner_microstep: 212.53 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 161.24 | bwd_microstep: 218.36 | bwd_inner_microstep: 218.07 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 160.73 | bwd_microstep: 212.33 | bwd_inner_microstep: 211.95 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 169.94 | bwd_microstep: 222.07 | bwd_inner_microstep: 221.72 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 150.53 | bwd_microstep: 222.37 | bwd_inner_microstep: 212.71 | bwd_allreduce_microstep: 9.59 | step_microstep: 21.24 [Rank 0] time (ms) | fwd: 1343.64 | bwd: 1736.59 | bwd_inner: 1724.96 | bwd_allreduce: 10.50 | step: 22.10 6%|▌ | 159/2865 [1:02:49<6:41:14, 8.90s/it] {'loss': 1.026, 'learning_rate': 3.993193530795747e-05, 'epoch': 0.17} 6%|▌ | 159/2865 [1:02:49<6:41:14, 8.90s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 946 [Rank 0] time (ms) | fwd_microstep: 161.88 | bwd_microstep: 216.95 | bwd_inner_microstep: 216.82 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.39 | bwd_microstep: 221.68 | bwd_inner_microstep: 221.41 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.18 | bwd_microstep: 215.43 | bwd_inner_microstep: 215.27 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 161.19 | bwd_microstep: 229.12 | bwd_inner_microstep: 229.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 162.14 | bwd_microstep: 229.11 | bwd_inner_microstep: 228.75 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 175.21 | bwd_microstep: 219.07 | bwd_inner_microstep: 218.86 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 165.47 | bwd_microstep: 223.71 | bwd_inner_microstep: 223.53 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 148.94 | bwd_microstep: 230.59 | bwd_inner_microstep: 220.00 | bwd_allreduce_microstep: 10.43 | step_microstep: 21.35 [Rank 0] time (ms) | fwd: 1295.30 | bwd: 1785.76 | bwd_inner: 1774.00 | bwd_allreduce: 10.99 | step: 22.03 6%|▌ | 160/2865 [1:02:52<5:24:42, 7.20s/it] {'loss': 0.9734, 'learning_rate': 3.9930058844444365e-05, 'epoch': 0.17} 6%|▌ | 160/2865 [1:02:52<5:24:42, 7.20s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 161.95 | bwd_microstep: 218.51 | bwd_inner_microstep: 218.37 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 160.81 | bwd_microstep: 215.74 | bwd_inner_microstep: 215.37 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 168.53 | bwd_microstep: 219.08 | bwd_inner_microstep: 218.85 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 165.01 | bwd_microstep: 219.56 | bwd_inner_microstep: 219.43 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.53 | bwd_microstep: 218.18 | bwd_inner_microstep: 217.81 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 161.13 | bwd_microstep: 219.86 | bwd_inner_microstep: 219.68 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 161.34 | bwd_microstep: 218.53 | bwd_inner_microstep: 218.30 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.16 | bwd_microstep: 256.40 | bwd_inner_microstep: 244.79 | bwd_allreduce_microstep: 11.43 | step_microstep: 21.24 [Rank 0] time (ms) | fwd: 1300.33 | bwd: 1785.99 | bwd_inner: 1773.04 | bwd_allreduce: 12.03 | step: 22.00 6%|▌ | 161/2865 [1:02:56<4:31:14, 6.02s/it] {'loss': 0.9314, 'learning_rate': 3.99281569107998e-05, 'epoch': 0.17} 6%|▌ | 161/2865 [1:02:56<4:31:14, 6.02s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 162.23 | bwd_microstep: 225.57 | bwd_inner_microstep: 225.30 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 175.17 | bwd_microstep: 225.81 | bwd_inner_microstep: 225.60 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.05 | bwd_microstep: 220.01 | bwd_inner_microstep: 219.64 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 163.71 | bwd_microstep: 246.22 | bwd_inner_microstep: 246.02 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.04 | bwd_microstep: 220.62 | bwd_inner_microstep: 220.36 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 162.81 | bwd_microstep: 216.79 | bwd_inner_microstep: 216.55 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 162.33 | bwd_microstep: 228.22 | bwd_inner_microstep: 227.91 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 148.65 | bwd_microstep: 224.20 | bwd_inner_microstep: 214.30 | bwd_allreduce_microstep: 9.79 | step_microstep: 21.39 [Rank 0] time (ms) | fwd: 1296.86 | bwd: 1807.55 | bwd_inner: 1796.13 | bwd_allreduce: 10.51 | step: 22.13 6%|▌ | 162/2865 [1:02:59<3:54:08, 5.20s/it] {'loss': 1.0541, 'learning_rate': 3.9926229509454393e-05, 'epoch': 0.17} 6%|▌ | 162/2865 [1:02:59<3:54:08, 5.20s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 162.14 | bwd_microstep: 217.97 | bwd_inner_microstep: 217.81 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 164.04 | bwd_microstep: 219.10 | bwd_inner_microstep: 218.73 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 161.11 | bwd_microstep: 244.02 | bwd_inner_microstep: 243.72 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.90 | bwd_microstep: 251.33 | bwd_inner_microstep: 250.98 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.88 | bwd_microstep: 217.84 | bwd_inner_microstep: 217.44 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 184.29 | bwd_microstep: 222.07 | bwd_inner_microstep: 221.73 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 161.62 | bwd_microstep: 223.76 | bwd_inner_microstep: 223.38 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.46 | bwd_microstep: 253.57 | bwd_inner_microstep: 243.64 | bwd_allreduce_microstep: 9.77 | step_microstep: 21.18 [Rank 0] time (ms) | fwd: 1317.31 | bwd: 1849.79 | bwd_inner: 1837.99 | bwd_allreduce: 10.64 | step: 22.01 6%|▌ | 163/2865 [1:03:02<3:29:08, 4.64s/it] {'loss': 1.1284, 'learning_rate': 3.9924276642871316e-05, 'epoch': 0.17} 6%|▌ | 163/2865 [1:03:02<3:29:08, 4.64s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 941 [Rank 0] time (ms) | fwd_microstep: 161.71 | bwd_microstep: 216.40 | bwd_inner_microstep: 216.23 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 175.54 | bwd_microstep: 225.15 | bwd_inner_microstep: 224.83 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 169.21 | bwd_microstep: 221.18 | bwd_inner_microstep: 220.83 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 165.55 | bwd_microstep: 220.17 | bwd_inner_microstep: 219.93 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 169.68 | bwd_microstep: 220.03 | bwd_inner_microstep: 219.76 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 169.01 | bwd_microstep: 219.28 | bwd_inner_microstep: 218.95 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 162.23 | bwd_microstep: 216.66 | bwd_inner_microstep: 216.36 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.87 | bwd_microstep: 217.91 | bwd_inner_microstep: 208.09 | bwd_allreduce_microstep: 9.69 | step_microstep: 21.27 [Rank 0] time (ms) | fwd: 1335.68 | bwd: 1756.91 | bwd_inner: 1745.48 | bwd_allreduce: 10.50 | step: 21.97 6%|▌ | 164/2865 [1:03:06<3:10:41, 4.24s/it] {'loss': 1.168, 'learning_rate': 3.992229831354629e-05, 'epoch': 0.17} 6%|▌ | 164/2865 [1:03:06<3:10:41, 4.24s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.86 | bwd_microstep: 190.45 | bwd_inner_microstep: 190.10 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 160.84 | bwd_microstep: 191.18 | bwd_inner_microstep: 190.81 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.78 | bwd_microstep: 190.37 | bwd_inner_microstep: 190.24 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 162.28 | bwd_microstep: 190.67 | bwd_inner_microstep: 190.37 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 162.43 | bwd_microstep: 212.43 | bwd_inner_microstep: 212.30 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 161.27 | bwd_microstep: 190.67 | bwd_inner_microstep: 190.54 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.52 | bwd_microstep: 190.80 | bwd_inner_microstep: 190.66 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.53 | bwd_microstep: 223.73 | bwd_inner_microstep: 213.94 | bwd_allreduce_microstep: 9.63 | step_microstep: 20.97 [Rank 0] time (ms) | fwd: 1289.40 | bwd: 1580.42 | bwd_inner: 1569.37 | bwd_allreduce: 10.24 | step: 21.59 6%|▌ | 165/2865 [1:03:09<2:54:22, 3.88s/it] {'loss': 1.0302, 'learning_rate': 3.9920294524007576e-05, 'epoch': 0.17} 6%|▌ | 165/2865 [1:03:09<2:54:22, 3.88s/it][INFO|trainer.py:3242] 2026-05-14 21:23:51,734 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 21:23:51,734 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 21:23:51,735 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-165 [INFO|configuration_utils.py:473] 2026-05-14 21:28:56,579 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-165/config.json [INFO|configuration_utils.py:594] 2026-05-14 21:28:56,587 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-165/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 21:29:01,826 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-165/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 21:29:01,838 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-165/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 21:29:01,848 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-165/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 21:29:01,851 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-165/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 21:29:11,471 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-90] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 302082.70 | bwd_microstep: 247.48 | bwd_inner_microstep: 247.05 | bwd_allreduce_microstep: 0.21 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 169.26 | bwd_microstep: 221.21 | bwd_inner_microstep: 221.02 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 175.70 | bwd_microstep: 220.79 | bwd_inner_microstep: 220.66 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.47 | bwd_microstep: 225.14 | bwd_inner_microstep: 224.79 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.29 | bwd_microstep: 215.84 | bwd_inner_microstep: 215.71 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.51 | bwd_microstep: 219.83 | bwd_inner_microstep: 219.44 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.25 | bwd_microstep: 221.75 | bwd_inner_microstep: 221.38 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.94 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 147.36 | bwd_microstep: 223.80 | bwd_inner_microstep: 213.57 | bwd_allreduce_microstep: 10.07 | step_microstep: 21.50 [Rank 0] time (ms) | fwd: 303210.52 | bwd: 1796.05 | bwd_inner: 1784.02 | bwd_allreduce: 10.94 | step: 22.41 6%|▌ | 166/2865 [1:08:32<74:48:33, 99.78s/it] {'loss': 0.9554, 'learning_rate': 3.991826527681597e-05, 'epoch': 0.17} 6%|▌ | 166/2865 [1:08:32<74:48:33, 99.78s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 161.77 | bwd_microstep: 217.98 | bwd_inner_microstep: 217.86 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 162.41 | bwd_microstep: 217.96 | bwd_inner_microstep: 217.82 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.10 | bwd_microstep: 218.40 | bwd_inner_microstep: 218.13 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.22 | bwd_microstep: 220.75 | bwd_inner_microstep: 220.61 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.38 | bwd_microstep: 213.75 | bwd_inner_microstep: 213.63 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 159.88 | bwd_microstep: 223.68 | bwd_inner_microstep: 223.34 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 162.98 | bwd_microstep: 221.56 | bwd_inner_microstep: 221.41 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 159.20 | bwd_microstep: 254.03 | bwd_inner_microstep: 244.13 | bwd_allreduce_microstep: 9.75 | step_microstep: 21.14 [Rank 0] time (ms) | fwd: 1289.83 | bwd: 1788.21 | bwd_inner: 1777.27 | bwd_allreduce: 10.23 | step: 21.67 6%|▌ | 167/2865 [1:08:35<53:04:33, 70.82s/it] {'loss': 1.0582, 'learning_rate': 3.9916210574564786e-05, 'epoch': 0.17} 6%|▌ | 167/2865 [1:08:35<53:04:33, 70.82s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 161.28 | bwd_microstep: 219.00 | bwd_inner_microstep: 218.73 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 165.35 | bwd_microstep: 218.45 | bwd_inner_microstep: 218.32 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 161.59 | bwd_microstep: 225.73 | bwd_inner_microstep: 225.38 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.54 | bwd_microstep: 222.22 | bwd_inner_microstep: 221.87 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 168.84 | bwd_microstep: 225.98 | bwd_inner_microstep: 225.78 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 169.20 | bwd_microstep: 218.18 | bwd_inner_microstep: 217.98 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.17 | bwd_microstep: 245.83 | bwd_inner_microstep: 245.73 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 147.62 | bwd_microstep: 226.73 | bwd_inner_microstep: 217.01 | bwd_allreduce_microstep: 9.65 | step_microstep: 21.27 [Rank 0] time (ms) | fwd: 1296.48 | bwd: 1802.26 | bwd_inner: 1791.12 | bwd_allreduce: 10.33 | step: 22.07 6%|▌ | 168/2865 [1:08:39<37:52:41, 50.56s/it] {'loss': 1.1937, 'learning_rate': 3.991413041987989e-05, 'epoch': 0.18} 6%|▌ | 168/2865 [1:08:39<37:52:41, 50.56s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 161.53 | bwd_microstep: 219.44 | bwd_inner_microstep: 219.12 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 162.26 | bwd_microstep: 223.52 | bwd_inner_microstep: 223.13 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 161.09 | bwd_microstep: 226.31 | bwd_inner_microstep: 226.17 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.99 | bwd_microstep: 224.34 | bwd_inner_microstep: 224.20 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 160.19 | bwd_microstep: 222.59 | bwd_inner_microstep: 222.30 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.10 | bwd_microstep: 219.78 | bwd_inner_microstep: 219.65 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.78 | bwd_microstep: 219.50 | bwd_inner_microstep: 219.23 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 146.68 | bwd_microstep: 224.90 | bwd_inner_microstep: 213.16 | bwd_allreduce_microstep: 11.67 | step_microstep: 21.26 [Rank 0] time (ms) | fwd: 1276.51 | bwd: 1780.51 | bwd_inner: 1767.28 | bwd_allreduce: 12.36 | step: 21.92 6%|▌ | 169/2865 [1:08:42<27:13:47, 36.36s/it] {'loss': 0.8932, 'learning_rate': 3.991202481541968e-05, 'epoch': 0.18} 6%|▌ | 169/2865 [1:08:42<27:13:47, 36.36s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 161.00 | bwd_microstep: 216.11 | bwd_inner_microstep: 215.81 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.22 | bwd_microstep: 218.46 | bwd_inner_microstep: 218.12 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 160.51 | bwd_microstep: 215.01 | bwd_inner_microstep: 214.63 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.08 | bwd_microstep: 217.70 | bwd_inner_microstep: 217.18 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 160.66 | bwd_microstep: 219.75 | bwd_inner_microstep: 219.44 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 162.00 | bwd_microstep: 217.65 | bwd_inner_microstep: 217.36 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 159.95 | bwd_microstep: 221.10 | bwd_inner_microstep: 220.99 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.16 | bwd_microstep: 253.42 | bwd_inner_microstep: 243.20 | bwd_allreduce_microstep: 10.11 | step_microstep: 21.08 [Rank 0] time (ms) | fwd: 1285.47 | bwd: 1779.32 | bwd_inner: 1767.13 | bwd_allreduce: 11.30 | step: 21.73 6%|▌ | 170/2865 [1:08:45<19:46:39, 26.42s/it] {'loss': 1.1288, 'learning_rate': 3.990989376387505e-05, 'epoch': 0.18} 6%|▌ | 170/2865 [1:08:45<19:46:39, 26.42s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 161.32 | bwd_microstep: 219.81 | bwd_inner_microstep: 219.45 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 161.95 | bwd_microstep: 254.32 | bwd_inner_microstep: 254.11 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.13 | bwd_microstep: 218.23 | bwd_inner_microstep: 218.10 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.14 | bwd_microstep: 216.69 | bwd_inner_microstep: 216.46 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.21 | bwd_microstep: 219.70 | bwd_inner_microstep: 219.39 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.02 | bwd_microstep: 218.68 | bwd_inner_microstep: 218.35 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 159.83 | bwd_microstep: 218.76 | bwd_inner_microstep: 218.39 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 160.76 | bwd_microstep: 247.77 | bwd_inner_microstep: 238.15 | bwd_allreduce_microstep: 9.55 | step_microstep: 21.04 [Rank 0] time (ms) | fwd: 1284.26 | bwd: 1814.13 | bwd_inner: 1802.79 | bwd_allreduce: 10.35 | step: 21.88 6%|▌ | 171/2865 [1:08:48<14:34:16, 19.47s/it] {'loss': 0.7828, 'learning_rate': 3.990773726796944e-05, 'epoch': 0.18} 6%|▌ | 171/2865 [1:08:48<14:34:16, 19.47s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 160.55 | bwd_microstep: 220.08 | bwd_inner_microstep: 219.67 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 158.50 | bwd_microstep: 220.78 | bwd_inner_microstep: 220.44 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 157.73 | bwd_microstep: 237.29 | bwd_inner_microstep: 236.92 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 159.39 | bwd_microstep: 221.13 | bwd_inner_microstep: 220.76 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 160.50 | bwd_microstep: 222.02 | bwd_inner_microstep: 221.91 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 158.12 | bwd_microstep: 222.19 | bwd_inner_microstep: 222.08 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 155.50 | bwd_microstep: 235.17 | bwd_inner_microstep: 235.06 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 155.98 | bwd_microstep: 228.75 | bwd_inner_microstep: 218.60 | bwd_allreduce_microstep: 9.99 | step_microstep: 21.12 [Rank 0] time (ms) | fwd: 1266.15 | bwd: 1807.58 | bwd_inner: 1795.85 | bwd_allreduce: 10.74 | step: 22.08 6%|▌ | 172/2865 [1:08:52<10:55:26, 14.60s/it] {'loss': 1.1117, 'learning_rate': 3.99055553304588e-05, 'epoch': 0.18} 6%|▌ | 172/2865 [1:08:52<10:55:26, 14.60s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 156.74 | bwd_microstep: 209.05 | bwd_inner_microstep: 208.72 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 158.37 | bwd_microstep: 216.76 | bwd_inner_microstep: 216.67 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 156.15 | bwd_microstep: 218.35 | bwd_inner_microstep: 218.23 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 156.36 | bwd_microstep: 218.11 | bwd_inner_microstep: 217.79 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 158.10 | bwd_microstep: 217.99 | bwd_inner_microstep: 217.80 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 158.54 | bwd_microstep: 219.80 | bwd_inner_microstep: 219.57 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.45 | bwd_microstep: 195.07 | bwd_inner_microstep: 194.71 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.79 | bwd_microstep: 223.78 | bwd_inner_microstep: 214.00 | bwd_allreduce_microstep: 9.67 | step_microstep: 21.32 [Rank 0] time (ms) | fwd: 1266.40 | bwd: 1719.00 | bwd_inner: 1707.93 | bwd_allreduce: 10.28 | step: 21.88 6%|▌ | 173/2865 [1:08:55<8:21:11, 11.17s/it] {'loss': 0.8938, 'learning_rate': 3.9903347954131607e-05, 'epoch': 0.18} 6%|▌ | 173/2865 [1:08:55<8:21:11, 11.17s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 175.55 | bwd_microstep: 208.49 | bwd_inner_microstep: 208.12 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 164.85 | bwd_microstep: 218.45 | bwd_inner_microstep: 218.04 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 160.73 | bwd_microstep: 219.82 | bwd_inner_microstep: 219.58 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 162.43 | bwd_microstep: 217.09 | bwd_inner_microstep: 216.82 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 162.59 | bwd_microstep: 217.18 | bwd_inner_microstep: 216.80 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 162.29 | bwd_microstep: 221.91 | bwd_inner_microstep: 221.55 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.72 | bwd_microstep: 214.19 | bwd_inner_microstep: 214.03 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 147.14 | bwd_microstep: 222.52 | bwd_inner_microstep: 212.80 | bwd_allreduce_microstep: 9.64 | step_microstep: 21.11 [Rank 0] time (ms) | fwd: 1296.18 | bwd: 1739.85 | bwd_inner: 1728.23 | bwd_allreduce: 10.46 | step: 21.99 6%|▌ | 174/2865 [1:08:58<6:33:52, 8.78s/it] {'loss': 1.4246, 'learning_rate': 3.9901115141808803e-05, 'epoch': 0.18} 6%|▌ | 174/2865 [1:08:58<6:33:52, 8.78s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 162.31 | bwd_microstep: 217.81 | bwd_inner_microstep: 217.49 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 162.06 | bwd_microstep: 220.89 | bwd_inner_microstep: 220.51 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 162.29 | bwd_microstep: 220.92 | bwd_inner_microstep: 220.77 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 161.18 | bwd_microstep: 223.18 | bwd_inner_microstep: 223.08 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 160.42 | bwd_microstep: 217.52 | bwd_inner_microstep: 217.22 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.48 | bwd_microstep: 218.78 | bwd_inner_microstep: 218.43 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 164.13 | bwd_microstep: 218.28 | bwd_inner_microstep: 217.91 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.10 | bwd_microstep: 249.37 | bwd_inner_microstep: 239.53 | bwd_allreduce_microstep: 9.69 | step_microstep: 21.10 [Rank 0] time (ms) | fwd: 1292.88 | bwd: 1786.89 | bwd_inner: 1775.35 | bwd_allreduce: 10.52 | step: 22.04 6%|▌ | 175/2865 [1:09:01<5:19:15, 7.12s/it] {'loss': 0.8903, 'learning_rate': 3.98988568963439e-05, 'epoch': 0.18} 6%|▌ | 175/2865 [1:09:01<5:19:15, 7.12s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 161.51 | bwd_microstep: 213.82 | bwd_inner_microstep: 213.44 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 164.07 | bwd_microstep: 218.73 | bwd_inner_microstep: 218.38 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 160.58 | bwd_microstep: 221.42 | bwd_inner_microstep: 221.03 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 161.11 | bwd_microstep: 218.18 | bwd_inner_microstep: 218.06 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 165.25 | bwd_microstep: 221.16 | bwd_inner_microstep: 220.56 | bwd_allreduce_microstep: 0.25 | step_microstep: 0.28 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.27 | bwd_microstep: 216.52 | bwd_inner_microstep: 216.40 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 159.80 | bwd_microstep: 217.95 | bwd_inner_microstep: 217.62 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 166.49 | bwd_microstep: 250.48 | bwd_inner_microstep: 240.11 | bwd_allreduce_microstep: 10.20 | step_microstep: 21.30 [Rank 0] time (ms) | fwd: 1298.98 | bwd: 1778.48 | bwd_inner: 1766.09 | bwd_allreduce: 11.18 | step: 22.30 6%|▌ | 176/2865 [1:09:04<4:26:59, 5.96s/it] {'loss': 1.1955, 'learning_rate': 3.9896573220622856e-05, 'epoch': 0.18} 6%|▌ | 176/2865 [1:09:04<4:26:59, 5.96s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 162.39 | bwd_microstep: 219.20 | bwd_inner_microstep: 218.88 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 163.73 | bwd_microstep: 221.96 | bwd_inner_microstep: 221.57 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.90 | bwd_microstep: 221.44 | bwd_inner_microstep: 221.20 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.77 | bwd_microstep: 247.38 | bwd_inner_microstep: 247.09 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 162.01 | bwd_microstep: 216.44 | bwd_inner_microstep: 216.18 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.63 | bwd_microstep: 217.93 | bwd_inner_microstep: 217.78 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 160.98 | bwd_microstep: 220.12 | bwd_inner_microstep: 219.80 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.44 | bwd_microstep: 255.26 | bwd_inner_microstep: 245.46 | bwd_allreduce_microstep: 9.71 | step_microstep: 21.17 [Rank 0] time (ms) | fwd: 1295.74 | bwd: 1819.82 | bwd_inner: 1808.41 | bwd_allreduce: 10.50 | step: 21.78 6%|▌ | 177/2865 [1:09:08<3:50:58, 5.16s/it] {'loss': 1.145, 'learning_rate': 3.9894264117564165e-05, 'epoch': 0.19} 6%|▌ | 177/2865 [1:09:08<3:50:58, 5.16s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 161.61 | bwd_microstep: 222.34 | bwd_inner_microstep: 222.09 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 172.64 | bwd_microstep: 222.12 | bwd_inner_microstep: 221.83 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 172.46 | bwd_microstep: 219.16 | bwd_inner_microstep: 218.95 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 172.42 | bwd_microstep: 219.70 | bwd_inner_microstep: 219.54 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 164.40 | bwd_microstep: 221.91 | bwd_inner_microstep: 221.47 | bwd_allreduce_microstep: 0.21 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 161.92 | bwd_microstep: 219.48 | bwd_inner_microstep: 219.24 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 161.99 | bwd_microstep: 218.27 | bwd_inner_microstep: 217.93 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 161.96 | bwd_microstep: 260.66 | bwd_inner_microstep: 250.61 | bwd_allreduce_microstep: 9.91 | step_microstep: 21.45 [Rank 0] time (ms) | fwd: 1329.27 | bwd: 1803.77 | bwd_inner: 1792.12 | bwd_allreduce: 10.67 | step: 22.14 6%|▌ | 178/2865 [1:09:11<3:26:17, 4.61s/it] {'loss': 0.932, 'learning_rate': 3.989192959011881e-05, 'epoch': 0.19} 6%|▌ | 178/2865 [1:09:11<3:26:17, 4.61s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 162.22 | bwd_microstep: 243.05 | bwd_inner_microstep: 242.74 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 161.60 | bwd_microstep: 227.03 | bwd_inner_microstep: 226.67 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.02 | bwd_microstep: 224.26 | bwd_inner_microstep: 224.12 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.46 | bwd_microstep: 226.93 | bwd_inner_microstep: 226.71 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 160.52 | bwd_microstep: 236.67 | bwd_inner_microstep: 236.55 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.98 | bwd_microstep: 219.99 | bwd_inner_microstep: 219.77 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 168.21 | bwd_microstep: 244.70 | bwd_inner_microstep: 244.33 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.73 | bwd_microstep: 281.38 | bwd_inner_microstep: 271.28 | bwd_allreduce_microstep: 9.94 | step_microstep: 21.20 [Rank 0] time (ms) | fwd: 1298.62 | bwd: 1904.17 | bwd_inner: 1892.57 | bwd_allreduce: 10.66 | step: 22.02 6%|▌ | 179/2865 [1:09:14<3:09:34, 4.23s/it] {'loss': 1.0969, 'learning_rate': 3.988956964127025e-05, 'epoch': 0.19} 6%|▌ | 179/2865 [1:09:14<3:09:34, 4.23s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 947 [Rank 0] time (ms) | fwd_microstep: 162.77 | bwd_microstep: 218.82 | bwd_inner_microstep: 218.70 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 160.71 | bwd_microstep: 246.70 | bwd_inner_microstep: 246.48 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.49 | bwd_microstep: 249.31 | bwd_inner_microstep: 249.01 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 160.82 | bwd_microstep: 232.09 | bwd_inner_microstep: 231.97 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 160.90 | bwd_microstep: 233.30 | bwd_inner_microstep: 232.94 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 146.72 | bwd_microstep: 191.31 | bwd_inner_microstep: 190.97 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.44 | bwd_microstep: 219.35 | bwd_inner_microstep: 219.14 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.01 | bwd_microstep: 254.25 | bwd_inner_microstep: 244.52 | bwd_allreduce_microstep: 9.65 | step_microstep: 21.42 [Rank 0] time (ms) | fwd: 1274.76 | bwd: 1845.24 | bwd_inner: 1834.16 | bwd_allreduce: 10.32 | step: 22.06 6%|▋ | 180/2865 [1:09:18<2:56:42, 3.95s/it] {'loss': 1.0822, 'learning_rate': 3.988718427403444e-05, 'epoch': 0.19} 6%|▋ | 180/2865 [1:09:18<2:56:42, 3.95s/it][INFO|trainer.py:3242] 2026-05-14 21:30:00,911 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 21:30:00,912 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 21:30:00,912 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-180 [INFO|configuration_utils.py:473] 2026-05-14 21:35:06,088 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-180/config.json [INFO|configuration_utils.py:594] 2026-05-14 21:35:06,095 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-180/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 21:35:11,523 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-180/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 21:35:11,533 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-180/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 21:35:11,539 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-180/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 21:35:11,541 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-180/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 21:35:21,328 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-105] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 947 [Rank 0] time (ms) | fwd_microstep: 302125.87 | bwd_microstep: 246.04 | bwd_inner_microstep: 245.86 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.64 | bwd_microstep: 234.18 | bwd_inner_microstep: 233.97 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 166.24 | bwd_microstep: 238.58 | bwd_inner_microstep: 238.19 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.76 | bwd_microstep: 220.69 | bwd_inner_microstep: 220.31 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 161.63 | bwd_microstep: 221.51 | bwd_inner_microstep: 221.29 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 162.71 | bwd_microstep: 228.69 | bwd_inner_microstep: 228.56 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.73 | bwd_microstep: 219.55 | bwd_inner_microstep: 219.19 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.44 | bwd_microstep: 249.82 | bwd_inner_microstep: 239.73 | bwd_allreduce_microstep: 9.94 | step_microstep: 21.02 [Rank 0] time (ms) | fwd: 303252.51 | bwd: 1859.23 | bwd_inner: 1847.51 | bwd_allreduce: 10.69 | step: 21.91 6%|▋ | 181/2865 [1:14:42<74:36:18, 100.07s/it] {'loss': 1.124, 'learning_rate': 3.988477349145983e-05, 'epoch': 0.19} 6%|▋ | 181/2865 [1:14:42<74:36:18, 100.07s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.87 | bwd_microstep: 242.82 | bwd_inner_microstep: 242.68 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.11 | bwd_microstep: 214.65 | bwd_inner_microstep: 214.26 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.12 | bwd_microstep: 219.80 | bwd_inner_microstep: 219.47 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.98 | bwd_microstep: 222.09 | bwd_inner_microstep: 221.82 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.35 | bwd_microstep: 216.02 | bwd_inner_microstep: 215.64 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.15 | bwd_microstep: 216.30 | bwd_inner_microstep: 216.09 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 161.06 | bwd_microstep: 225.74 | bwd_inner_microstep: 225.35 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.25 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 159.96 | bwd_microstep: 251.11 | bwd_inner_microstep: 240.48 | bwd_allreduce_microstep: 10.55 | step_microstep: 25.65 [Rank 0] time (ms) | fwd: 1288.47 | bwd: 1808.72 | bwd_inner: 1796.29 | bwd_allreduce: 11.42 | step: 26.51 6%|▋ | 182/2865 [1:14:45<52:56:18, 71.03s/it] {'loss': 1.0855, 'learning_rate': 3.988233729662735e-05, 'epoch': 0.19} 6%|▋ | 182/2865 [1:14:45<52:56:18, 71.03s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 161.54 | bwd_microstep: 218.46 | bwd_inner_microstep: 218.33 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 160.99 | bwd_microstep: 218.60 | bwd_inner_microstep: 218.23 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 160.70 | bwd_microstep: 219.83 | bwd_inner_microstep: 219.60 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 161.70 | bwd_microstep: 221.46 | bwd_inner_microstep: 221.22 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 182.93 | bwd_microstep: 217.95 | bwd_inner_microstep: 217.66 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 162.09 | bwd_microstep: 221.41 | bwd_inner_microstep: 221.01 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 162.06 | bwd_microstep: 218.05 | bwd_inner_microstep: 217.91 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.21 | bwd_microstep: 252.53 | bwd_inner_microstep: 240.82 | bwd_allreduce_microstep: 11.47 | step_microstep: 21.29 [Rank 0] time (ms) | fwd: 1314.12 | bwd: 1788.42 | bwd_inner: 1775.28 | bwd_allreduce: 12.12 | step: 21.97 6%|▋ | 183/2865 [1:14:49<37:46:35, 50.71s/it] {'loss': 1.1742, 'learning_rate': 3.987987569265037e-05, 'epoch': 0.19} 6%|▋ | 183/2865 [1:14:49<37:46:35, 50.71s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 163.79 | bwd_microstep: 220.73 | bwd_inner_microstep: 220.38 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 162.82 | bwd_microstep: 219.39 | bwd_inner_microstep: 219.08 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 163.54 | bwd_microstep: 226.36 | bwd_inner_microstep: 226.06 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 165.46 | bwd_microstep: 219.93 | bwd_inner_microstep: 219.71 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.70 | bwd_microstep: 221.94 | bwd_inner_microstep: 221.69 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 163.61 | bwd_microstep: 224.52 | bwd_inner_microstep: 224.13 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 161.48 | bwd_microstep: 236.20 | bwd_inner_microstep: 235.89 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.06 | bwd_microstep: 254.66 | bwd_inner_microstep: 244.65 | bwd_allreduce_microstep: 9.82 | step_microstep: 21.51 [Rank 0] time (ms) | fwd: 1303.32 | bwd: 1823.83 | bwd_inner: 1812.15 | bwd_allreduce: 10.62 | step: 22.17 6%|▋ | 184/2865 [1:14:52<27:10:26, 36.49s/it] {'loss': 1.191, 'learning_rate': 3.9877388682674795e-05, 'epoch': 0.19} 6%|▋ | 184/2865 [1:14:52<27:10:26, 36.49s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 161.55 | bwd_microstep: 218.43 | bwd_inner_microstep: 217.93 | bwd_allreduce_microstep: 0.21 | step_microstep: 0.30 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.61 | bwd_microstep: 221.87 | bwd_inner_microstep: 221.74 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 161.94 | bwd_microstep: 220.71 | bwd_inner_microstep: 220.33 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 162.53 | bwd_microstep: 221.39 | bwd_inner_microstep: 221.02 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 162.47 | bwd_microstep: 231.22 | bwd_inner_microstep: 230.88 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.31 | bwd_microstep: 221.02 | bwd_inner_microstep: 220.72 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 163.74 | bwd_microstep: 223.56 | bwd_inner_microstep: 223.30 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 147.83 | bwd_microstep: 230.52 | bwd_inner_microstep: 219.31 | bwd_allreduce_microstep: 11.06 | step_microstep: 21.53 [Rank 0] time (ms) | fwd: 1282.87 | bwd: 1788.94 | bwd_inner: 1775.74 | bwd_allreduce: 11.95 | step: 22.54 6%|▋ | 185/2865 [1:14:55<19:44:17, 26.51s/it] {'loss': 1.0664, 'learning_rate': 3.9874876269878935e-05, 'epoch': 0.19} 6%|▋ | 185/2865 [1:14:55<19:44:17, 26.51s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 176.80 | bwd_microstep: 223.12 | bwd_inner_microstep: 222.95 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 165.79 | bwd_microstep: 221.45 | bwd_inner_microstep: 221.07 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.66 | bwd_microstep: 221.20 | bwd_inner_microstep: 220.82 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 161.15 | bwd_microstep: 223.87 | bwd_inner_microstep: 223.65 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 146.78 | bwd_microstep: 220.25 | bwd_inner_microstep: 219.86 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.38 | bwd_microstep: 234.17 | bwd_inner_microstep: 234.04 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 146.66 | bwd_microstep: 219.00 | bwd_inner_microstep: 218.87 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.06 | bwd_microstep: 255.52 | bwd_inner_microstep: 243.45 | bwd_allreduce_microstep: 11.91 | step_microstep: 21.37 [Rank 0] time (ms) | fwd: 1281.14 | bwd: 1818.73 | bwd_inner: 1805.15 | bwd_allreduce: 12.63 | step: 22.12 6%|▋ | 186/2865 [1:14:58<14:32:37, 19.54s/it] {'loss': 1.0231, 'learning_rate': 3.98723384574736e-05, 'epoch': 0.19} 6%|▋ | 186/2865 [1:14:58<14:32:37, 19.54s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 168.28 | bwd_microstep: 219.28 | bwd_inner_microstep: 219.15 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.93 | bwd_microstep: 220.41 | bwd_inner_microstep: 220.05 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 163.00 | bwd_microstep: 271.70 | bwd_inner_microstep: 271.39 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 164.51 | bwd_microstep: 256.19 | bwd_inner_microstep: 255.82 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 162.24 | bwd_microstep: 218.06 | bwd_inner_microstep: 217.91 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.36 | bwd_microstep: 224.91 | bwd_inner_microstep: 224.62 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 154.90 | bwd_microstep: 193.63 | bwd_inner_microstep: 193.48 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.67 | bwd_microstep: 254.47 | bwd_inner_microstep: 244.62 | bwd_allreduce_microstep: 9.72 | step_microstep: 21.45 [Rank 0] time (ms) | fwd: 1295.71 | bwd: 1858.80 | bwd_inner: 1847.44 | bwd_allreduce: 10.47 | step: 22.24 7%|▋ | 187/2865 [1:15:02<10:55:26, 14.68s/it] {'loss': 1.1294, 'learning_rate': 3.986977524870207e-05, 'epoch': 0.2} 7%|▋ | 187/2865 [1:15:02<10:55:26, 14.68s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 162.51 | bwd_microstep: 218.33 | bwd_inner_microstep: 218.20 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 161.41 | bwd_microstep: 221.88 | bwd_inner_microstep: 221.74 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.12 | bwd_microstep: 221.03 | bwd_inner_microstep: 220.75 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 161.69 | bwd_microstep: 219.92 | bwd_inner_microstep: 219.65 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 176.29 | bwd_microstep: 220.88 | bwd_inner_microstep: 220.72 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 175.36 | bwd_microstep: 218.61 | bwd_inner_microstep: 218.44 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 171.85 | bwd_microstep: 224.52 | bwd_inner_microstep: 224.34 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.93 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 168.00 | bwd_microstep: 262.83 | bwd_inner_microstep: 252.70 | bwd_allreduce_microstep: 10.05 | step_microstep: 22.26 [Rank 0] time (ms) | fwd: 1338.09 | bwd: 1808.09 | bwd_inner: 1796.86 | bwd_allreduce: 10.57 | step: 22.88 7%|▋ | 188/2865 [1:15:05<8:23:00, 11.27s/it] {'loss': 1.0238, 'learning_rate': 3.9867186646840036e-05, 'epoch': 0.2} 7%|▋ | 188/2865 [1:15:05<8:23:00, 11.27s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 162.36 | bwd_microstep: 218.58 | bwd_inner_microstep: 218.19 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 160.97 | bwd_microstep: 219.74 | bwd_inner_microstep: 219.62 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 161.38 | bwd_microstep: 221.00 | bwd_inner_microstep: 220.65 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 161.25 | bwd_microstep: 223.11 | bwd_inner_microstep: 222.75 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 162.87 | bwd_microstep: 228.29 | bwd_inner_microstep: 228.16 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.52 | bwd_microstep: 219.56 | bwd_inner_microstep: 219.13 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 147.33 | bwd_microstep: 188.07 | bwd_inner_microstep: 187.90 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.96 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 160.52 | bwd_microstep: 252.98 | bwd_inner_microstep: 242.14 | bwd_allreduce_microstep: 10.70 | step_microstep: 21.54 [Rank 0] time (ms) | fwd: 1278.09 | bwd: 1771.50 | bwd_inner: 1758.96 | bwd_allreduce: 11.58 | step: 22.43 7%|▋ | 189/2865 [1:15:08<6:35:05, 8.86s/it] {'loss': 1.0379, 'learning_rate': 3.986457265519568e-05, 'epoch': 0.2} 7%|▋ | 189/2865 [1:15:08<6:35:05, 8.86s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 162.84 | bwd_microstep: 218.85 | bwd_inner_microstep: 218.55 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.83 | bwd_microstep: 219.77 | bwd_inner_microstep: 219.63 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 162.12 | bwd_microstep: 215.83 | bwd_inner_microstep: 215.49 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.13 | bwd_microstep: 216.41 | bwd_inner_microstep: 216.28 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 161.57 | bwd_microstep: 216.73 | bwd_inner_microstep: 216.59 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.16 | bwd_microstep: 215.86 | bwd_inner_microstep: 215.58 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.51 | bwd_microstep: 215.63 | bwd_inner_microstep: 215.51 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 160.93 | bwd_microstep: 217.86 | bwd_inner_microstep: 208.11 | bwd_allreduce_microstep: 9.66 | step_microstep: 21.09 [Rank 0] time (ms) | fwd: 1291.98 | bwd: 1737.02 | bwd_inner: 1726.06 | bwd_allreduce: 10.29 | step: 21.62 7%|▋ | 190/2865 [1:15:12<5:19:06, 7.16s/it] {'loss': 0.7904, 'learning_rate': 3.986193327710963e-05, 'epoch': 0.2} 7%|▋ | 190/2865 [1:15:12<5:19:06, 7.16s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 162.00 | bwd_microstep: 188.54 | bwd_inner_microstep: 188.40 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.38 | bwd_microstep: 190.50 | bwd_inner_microstep: 190.19 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.33 | bwd_microstep: 190.65 | bwd_inner_microstep: 190.52 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 161.35 | bwd_microstep: 188.86 | bwd_inner_microstep: 188.48 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 162.33 | bwd_microstep: 189.01 | bwd_inner_microstep: 188.87 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 162.63 | bwd_microstep: 190.13 | bwd_inner_microstep: 190.01 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 162.29 | bwd_microstep: 192.39 | bwd_inner_microstep: 192.21 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.36 | bwd_microstep: 222.77 | bwd_inner_microstep: 212.92 | bwd_allreduce_microstep: 9.66 | step_microstep: 21.33 [Rank 0] time (ms) | fwd: 1294.57 | bwd: 1552.96 | bwd_inner: 1541.97 | bwd_allreduce: 10.24 | step: 21.97 7%|▋ | 191/2865 [1:15:15<4:23:34, 5.91s/it] {'loss': 1.2695, 'learning_rate': 3.985926851595493e-05, 'epoch': 0.2} 7%|▋ | 191/2865 [1:15:15<4:23:34, 5.91s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 942 [Rank 0] time (ms) | fwd_microstep: 164.57 | bwd_microstep: 194.54 | bwd_inner_microstep: 194.24 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 168.11 | bwd_microstep: 192.81 | bwd_inner_microstep: 192.59 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 166.07 | bwd_microstep: 191.99 | bwd_inner_microstep: 191.74 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 169.05 | bwd_microstep: 190.57 | bwd_inner_microstep: 190.27 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 162.61 | bwd_microstep: 197.77 | bwd_inner_microstep: 197.64 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 161.96 | bwd_microstep: 196.72 | bwd_inner_microstep: 196.34 | bwd_allreduce_microstep: 0.21 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.67 | bwd_microstep: 194.50 | bwd_inner_microstep: 194.11 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.03 | bwd_microstep: 222.73 | bwd_inner_microstep: 210.76 | bwd_allreduce_microstep: 11.82 | step_microstep: 21.56 [Rank 0] time (ms) | fwd: 1316.95 | bwd: 1581.74 | bwd_inner: 1568.12 | bwd_allreduce: 12.63 | step: 22.27 7%|▋ | 192/2865 [1:15:18<3:45:21, 5.06s/it] {'loss': 1.0359, 'learning_rate': 3.98565783751371e-05, 'epoch': 0.2} 7%|▋ | 192/2865 [1:15:18<3:45:21, 5.06s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 162.44 | bwd_microstep: 192.67 | bwd_inner_microstep: 192.28 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 162.18 | bwd_microstep: 193.82 | bwd_inner_microstep: 193.52 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.55 | bwd_microstep: 192.06 | bwd_inner_microstep: 191.91 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 164.62 | bwd_microstep: 191.25 | bwd_inner_microstep: 191.09 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 164.49 | bwd_microstep: 192.16 | bwd_inner_microstep: 192.02 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 161.24 | bwd_microstep: 190.89 | bwd_inner_microstep: 190.51 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.27 | bwd_microstep: 197.04 | bwd_inner_microstep: 196.67 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.72 | bwd_microstep: 228.08 | bwd_inner_microstep: 213.54 | bwd_allreduce_microstep: 14.42 | step_microstep: 21.16 [Rank 0] time (ms) | fwd: 1300.41 | bwd: 1578.13 | bwd_inner: 1561.94 | bwd_allreduce: 15.23 | step: 22.11 7%|▋ | 193/2865 [1:15:21<3:18:26, 4.46s/it] {'loss': 1.1448, 'learning_rate': 3.9853862858094057e-05, 'epoch': 0.2} 7%|▋ | 193/2865 [1:15:21<3:18:26, 4.46s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 162.36 | bwd_microstep: 190.25 | bwd_inner_microstep: 189.88 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 162.29 | bwd_microstep: 192.25 | bwd_inner_microstep: 192.04 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.71 | bwd_microstep: 195.20 | bwd_inner_microstep: 194.83 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 162.21 | bwd_microstep: 197.26 | bwd_inner_microstep: 197.02 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 163.80 | bwd_microstep: 222.74 | bwd_inner_microstep: 222.61 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 160.91 | bwd_microstep: 225.40 | bwd_inner_microstep: 225.03 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 161.12 | bwd_microstep: 215.20 | bwd_inner_microstep: 215.07 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.22 | bwd_microstep: 251.80 | bwd_inner_microstep: 242.03 | bwd_allreduce_microstep: 9.70 | step_microstep: 21.11 [Rank 0] time (ms) | fwd: 1295.49 | bwd: 1690.27 | bwd_inner: 1678.83 | bwd_allreduce: 10.48 | step: 22.07 7%|▋ | 194/2865 [1:15:24<3:01:06, 4.07s/it] {'loss': 1.3462, 'learning_rate': 3.9851121968296176e-05, 'epoch': 0.2} 7%|▋ | 194/2865 [1:15:24<3:01:06, 4.07s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 160.79 | bwd_microstep: 217.37 | bwd_inner_microstep: 216.99 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 163.64 | bwd_microstep: 215.90 | bwd_inner_microstep: 215.77 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 161.32 | bwd_microstep: 215.43 | bwd_inner_microstep: 215.11 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.95 | bwd_microstep: 222.30 | bwd_inner_microstep: 221.91 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 163.21 | bwd_microstep: 218.71 | bwd_inner_microstep: 218.58 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.27 | bwd_microstep: 232.16 | bwd_inner_microstep: 231.84 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 161.21 | bwd_microstep: 217.62 | bwd_inner_microstep: 217.03 | bwd_allreduce_microstep: 0.23 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.53 | bwd_microstep: 247.95 | bwd_inner_microstep: 238.01 | bwd_allreduce_microstep: 9.78 | step_microstep: 21.17 [Rank 0] time (ms) | fwd: 1292.79 | bwd: 1787.61 | bwd_inner: 1775.76 | bwd_allreduce: 10.74 | step: 22.01 7%|▋ | 195/2865 [1:15:27<2:50:00, 3.82s/it] {'loss': 0.9218, 'learning_rate': 3.984835570924624e-05, 'epoch': 0.2} 7%|▋ | 195/2865 [1:15:27<2:50:00, 3.82s/it][INFO|trainer.py:3242] 2026-05-14 21:36:10,246 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 21:36:10,246 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 21:36:10,246 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-195 [INFO|configuration_utils.py:473] 2026-05-14 21:41:15,384 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-195/config.json [INFO|configuration_utils.py:594] 2026-05-14 21:41:15,395 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-195/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 21:41:21,205 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-195/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 21:41:21,219 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-195/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 21:41:21,226 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-195/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 21:41:21,228 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-195/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 21:41:30,984 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-120] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 302174.84 | bwd_microstep: 224.08 | bwd_inner_microstep: 223.64 | bwd_allreduce_microstep: 0.21 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.75 | bwd_microstep: 222.24 | bwd_inner_microstep: 222.00 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 166.10 | bwd_microstep: 220.27 | bwd_inner_microstep: 219.95 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 159.49 | bwd_microstep: 222.91 | bwd_inner_microstep: 222.75 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.77 | bwd_microstep: 218.43 | bwd_inner_microstep: 218.28 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 146.65 | bwd_microstep: 195.23 | bwd_inner_microstep: 194.94 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.63 | bwd_microstep: 222.61 | bwd_inner_microstep: 222.20 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.33 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.15 | bwd_microstep: 255.02 | bwd_inner_microstep: 244.01 | bwd_allreduce_microstep: 10.85 | step_microstep: 23.00 [Rank 0] time (ms) | fwd: 303281.64 | bwd: 1780.93 | bwd_inner: 1768.20 | bwd_allreduce: 11.73 | step: 23.74 7%|▋ | 196/2865 [1:20:52<74:09:36, 100.03s/it] {'loss': 1.073, 'learning_rate': 3.9845564084479466e-05, 'epoch': 0.21} 7%|▋ | 196/2865 [1:20:52<74:09:36, 100.03s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 166.33 | bwd_microstep: 218.62 | bwd_inner_microstep: 218.35 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 165.28 | bwd_microstep: 226.28 | bwd_inner_microstep: 225.94 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 164.36 | bwd_microstep: 223.73 | bwd_inner_microstep: 223.32 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.28 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 167.45 | bwd_microstep: 225.22 | bwd_inner_microstep: 225.10 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 161.82 | bwd_microstep: 224.29 | bwd_inner_microstep: 223.98 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 162.25 | bwd_microstep: 218.85 | bwd_inner_microstep: 218.52 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.78 | bwd_microstep: 217.48 | bwd_inner_microstep: 217.31 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 164.90 | bwd_microstep: 248.89 | bwd_inner_microstep: 236.97 | bwd_allreduce_microstep: 11.78 | step_microstep: 21.35 [Rank 0] time (ms) | fwd: 1313.02 | bwd: 1803.51 | bwd_inner: 1789.94 | bwd_allreduce: 12.55 | step: 22.04 7%|▋ | 197/2865 [1:20:55<52:37:49, 71.02s/it] {'loss': 0.9905, 'learning_rate': 3.9842747097563477e-05, 'epoch': 0.21} 7%|▋ | 197/2865 [1:20:55<52:37:49, 71.02s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 942 [Rank 0] time (ms) | fwd_microstep: 163.02 | bwd_microstep: 221.27 | bwd_inner_microstep: 220.96 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 161.42 | bwd_microstep: 219.00 | bwd_inner_microstep: 218.87 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 163.52 | bwd_microstep: 218.51 | bwd_inner_microstep: 218.15 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.82 | bwd_microstep: 219.74 | bwd_inner_microstep: 219.35 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 164.36 | bwd_microstep: 219.51 | bwd_inner_microstep: 219.12 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 161.09 | bwd_microstep: 236.03 | bwd_inner_microstep: 235.63 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 160.34 | bwd_microstep: 219.05 | bwd_inner_microstep: 218.78 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.37 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.44 | bwd_microstep: 504.23 | bwd_inner_microstep: 493.20 | bwd_allreduce_microstep: 10.86 | step_microstep: 24.08 [Rank 0] time (ms) | fwd: 1294.88 | bwd: 2057.54 | bwd_inner: 2044.60 | bwd_allreduce: 11.76 | step: 25.18 7%|▋ | 198/2865 [1:20:58<37:36:36, 50.77s/it] {'loss': 0.9532, 'learning_rate': 3.983990475209832e-05, 'epoch': 0.21} 7%|▋ | 198/2865 [1:20:58<37:36:36, 50.77s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 166.10 | bwd_microstep: 217.72 | bwd_inner_microstep: 217.43 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 162.21 | bwd_microstep: 239.20 | bwd_inner_microstep: 238.91 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 163.24 | bwd_microstep: 220.70 | bwd_inner_microstep: 220.46 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 162.48 | bwd_microstep: 217.20 | bwd_inner_microstep: 216.91 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 161.28 | bwd_microstep: 216.74 | bwd_inner_microstep: 216.44 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.15 | bwd_microstep: 220.58 | bwd_inner_microstep: 220.11 | bwd_allreduce_microstep: 0.26 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.17 | bwd_microstep: 220.04 | bwd_inner_microstep: 219.90 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.73 | bwd_microstep: 249.25 | bwd_inner_microstep: 239.44 | bwd_allreduce_microstep: 9.73 | step_microstep: 21.18 [Rank 0] time (ms) | fwd: 1298.23 | bwd: 1801.55 | bwd_inner: 1790.10 | bwd_allreduce: 10.44 | step: 21.82 7%|▋ | 199/2865 [1:21:02<27:02:35, 36.52s/it] {'loss': 0.9368, 'learning_rate': 3.983703705171645e-05, 'epoch': 0.21} 7%|▋ | 199/2865 [1:21:02<27:02:35, 36.52s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 941 [Rank 0] time (ms) | fwd_microstep: 161.89 | bwd_microstep: 218.33 | bwd_inner_microstep: 217.95 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.34 | bwd_microstep: 220.27 | bwd_inner_microstep: 219.91 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 161.65 | bwd_microstep: 223.18 | bwd_inner_microstep: 222.76 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.81 | bwd_microstep: 218.20 | bwd_inner_microstep: 217.87 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 163.13 | bwd_microstep: 220.39 | bwd_inner_microstep: 220.07 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.20 | bwd_microstep: 219.51 | bwd_inner_microstep: 219.39 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 160.54 | bwd_microstep: 220.03 | bwd_inner_microstep: 219.83 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 224.77 | bwd_microstep: 247.67 | bwd_inner_microstep: 237.79 | bwd_allreduce_microstep: 9.72 | step_microstep: 21.26 [Rank 0] time (ms) | fwd: 1355.24 | bwd: 1787.74 | bwd_inner: 1776.03 | bwd_allreduce: 10.64 | step: 22.20 7%|▋ | 200/2865 [1:21:05<19:39:30, 26.56s/it] {'loss': 1.1203, 'learning_rate': 3.98341440000827e-05, 'epoch': 0.21} 7%|▋ | 200/2865 [1:21:05<19:39:30, 26.56s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 161.26 | bwd_microstep: 215.63 | bwd_inner_microstep: 215.37 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 161.86 | bwd_microstep: 215.81 | bwd_inner_microstep: 215.39 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.76 | bwd_microstep: 217.75 | bwd_inner_microstep: 217.53 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 162.63 | bwd_microstep: 215.30 | bwd_inner_microstep: 215.17 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.25 | bwd_microstep: 218.68 | bwd_inner_microstep: 218.56 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 909 [Rank 0] time (ms) | fwd_microstep: 147.01 | bwd_microstep: 193.36 | bwd_inner_microstep: 193.19 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 148.48 | bwd_microstep: 212.98 | bwd_inner_microstep: 212.74 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.41 | bwd_microstep: 252.78 | bwd_inner_microstep: 242.90 | bwd_allreduce_microstep: 9.73 | step_microstep: 21.22 [Rank 0] time (ms) | fwd: 1266.54 | bwd: 1742.40 | bwd_inner: 1731.43 | bwd_allreduce: 10.22 | step: 21.76 7%|▋ | 201/2865 [1:21:08<14:27:46, 19.54s/it] {'loss': 1.0358, 'learning_rate': 3.9831225600894335e-05, 'epoch': 0.21} 7%|▋ | 201/2865 [1:21:08<14:27:46, 19.54s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 164.56 | bwd_microstep: 218.90 | bwd_inner_microstep: 218.54 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 161.84 | bwd_microstep: 219.60 | bwd_inner_microstep: 219.24 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.14 | bwd_microstep: 218.82 | bwd_inner_microstep: 218.46 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 169.93 | bwd_microstep: 215.45 | bwd_inner_microstep: 215.31 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 163.08 | bwd_microstep: 218.92 | bwd_inner_microstep: 218.69 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 148.22 | bwd_microstep: 192.27 | bwd_inner_microstep: 192.14 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 163.81 | bwd_microstep: 225.84 | bwd_inner_microstep: 225.71 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 162.21 | bwd_microstep: 250.10 | bwd_inner_microstep: 239.96 | bwd_allreduce_microstep: 10.03 | step_microstep: 21.78 [Rank 0] time (ms) | fwd: 1294.67 | bwd: 1760.04 | bwd_inner: 1748.44 | bwd_allreduce: 10.74 | step: 22.58 7%|▋ | 202/2865 [1:21:11<10:50:15, 14.65s/it] {'loss': 0.995, 'learning_rate': 3.9828281857880994e-05, 'epoch': 0.21} 7%|▋ | 202/2865 [1:21:11<10:50:15, 14.65s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 162.74 | bwd_microstep: 225.70 | bwd_inner_microstep: 225.37 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 169.05 | bwd_microstep: 253.91 | bwd_inner_microstep: 253.57 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 162.17 | bwd_microstep: 233.49 | bwd_inner_microstep: 233.29 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 191.11 | bwd_microstep: 267.70 | bwd_inner_microstep: 267.54 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 155.23 | bwd_microstep: 255.44 | bwd_inner_microstep: 255.13 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 156.19 | bwd_microstep: 253.21 | bwd_inner_microstep: 252.90 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 173.83 | bwd_microstep: 290.31 | bwd_inner_microstep: 289.82 | bwd_allreduce_microstep: 0.20 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.95 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 171.86 | bwd_microstep: 295.24 | bwd_inner_microstep: 275.33 | bwd_allreduce_microstep: 19.72 | step_microstep: 21.36 [Rank 0] time (ms) | fwd: 1342.01 | bwd: 2075.16 | bwd_inner: 2053.47 | bwd_allreduce: 20.60 | step: 22.22 7%|▋ | 203/2865 [1:21:15<8:23:30, 11.35s/it] {'loss': 0.8318, 'learning_rate': 3.982531277480471e-05, 'epoch': 0.21} 7%|▋ | 203/2865 [1:21:15<8:23:30, 11.35s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 177.81 | bwd_microstep: 287.47 | bwd_inner_microstep: 287.19 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 173.65 | bwd_microstep: 288.99 | bwd_inner_microstep: 288.69 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 175.87 | bwd_microstep: 300.91 | bwd_inner_microstep: 300.63 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 175.17 | bwd_microstep: 295.63 | bwd_inner_microstep: 295.28 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 175.68 | bwd_microstep: 308.09 | bwd_inner_microstep: 307.77 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 166.43 | bwd_microstep: 259.01 | bwd_inner_microstep: 258.58 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 162.89 | bwd_microstep: 239.61 | bwd_inner_microstep: 239.23 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 163.27 | bwd_microstep: 330.86 | bwd_inner_microstep: 319.01 | bwd_allreduce_microstep: 11.68 | step_microstep: 20.90 [Rank 0] time (ms) | fwd: 1370.61 | bwd: 2310.69 | bwd_inner: 2296.99 | bwd_allreduce: 12.53 | step: 21.68 7%|▋ | 204/2865 [1:21:19<6:44:02, 9.11s/it] {'loss': 1.0344, 'learning_rate': 3.98223183554599e-05, 'epoch': 0.21} 7%|▋ | 204/2865 [1:21:19<6:44:02, 9.11s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 162.82 | bwd_microstep: 280.31 | bwd_inner_microstep: 280.15 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 173.64 | bwd_microstep: 273.12 | bwd_inner_microstep: 272.79 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 175.54 | bwd_microstep: 270.29 | bwd_inner_microstep: 270.14 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.90 | bwd_microstep: 214.79 | bwd_inner_microstep: 214.67 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 161.54 | bwd_microstep: 221.58 | bwd_inner_microstep: 221.39 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 170.75 | bwd_microstep: 251.98 | bwd_inner_microstep: 251.63 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 161.33 | bwd_microstep: 222.65 | bwd_inner_microstep: 222.33 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 160.89 | bwd_microstep: 253.71 | bwd_inner_microstep: 243.88 | bwd_allreduce_microstep: 9.65 | step_microstep: 20.95 [Rank 0] time (ms) | fwd: 1327.27 | bwd: 1988.59 | bwd_inner: 1977.36 | bwd_allreduce: 10.34 | step: 21.66 7%|▋ | 205/2865 [1:21:22<5:29:16, 7.43s/it] {'loss': 1.0772, 'learning_rate': 3.981929860367335e-05, 'epoch': 0.21} 7%|▋ | 205/2865 [1:21:22<5:29:16, 7.43s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 946 [Rank 0] time (ms) | fwd_microstep: 166.07 | bwd_microstep: 225.71 | bwd_inner_microstep: 225.37 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 161.05 | bwd_microstep: 230.43 | bwd_inner_microstep: 230.26 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.46 | bwd_microstep: 245.06 | bwd_inner_microstep: 244.94 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 162.55 | bwd_microstep: 224.34 | bwd_inner_microstep: 223.98 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.43 | bwd_microstep: 217.93 | bwd_inner_microstep: 217.50 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.35 | bwd_microstep: 215.53 | bwd_inner_microstep: 215.21 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 160.98 | bwd_microstep: 214.82 | bwd_inner_microstep: 214.69 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 146.69 | bwd_microstep: 220.39 | bwd_inner_microstep: 210.42 | bwd_allreduce_microstep: 9.81 | step_microstep: 21.07 [Rank 0] time (ms) | fwd: 1279.46 | bwd: 1794.34 | bwd_inner: 1782.80 | bwd_allreduce: 10.60 | step: 21.82 7%|▋ | 206/2865 [1:21:26<4:33:37, 6.17s/it] {'loss': 0.9885, 'learning_rate': 3.981625352330424e-05, 'epoch': 0.22} 7%|▋ | 206/2865 [1:21:26<4:33:37, 6.17s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 163.66 | bwd_microstep: 210.97 | bwd_inner_microstep: 210.63 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.00 | bwd_microstep: 220.28 | bwd_inner_microstep: 219.84 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.76 | bwd_microstep: 222.75 | bwd_inner_microstep: 222.53 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 169.08 | bwd_microstep: 213.93 | bwd_inner_microstep: 213.72 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 168.60 | bwd_microstep: 212.78 | bwd_inner_microstep: 212.42 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 165.62 | bwd_microstep: 216.44 | bwd_inner_microstep: 216.11 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.90 | bwd_microstep: 216.89 | bwd_inner_microstep: 216.71 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.41 | bwd_microstep: 251.53 | bwd_inner_microstep: 241.32 | bwd_allreduce_microstep: 10.06 | step_microstep: 21.00 [Rank 0] time (ms) | fwd: 1311.91 | bwd: 1765.73 | bwd_inner: 1753.76 | bwd_allreduce: 10.92 | step: 21.82 7%|▋ | 207/2865 [1:21:29<3:54:54, 5.30s/it] {'loss': 0.8655, 'learning_rate': 3.9813183118244106e-05, 'epoch': 0.22} 7%|▋ | 207/2865 [1:21:29<3:54:54, 5.30s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 163.27 | bwd_microstep: 214.49 | bwd_inner_microstep: 214.38 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 165.02 | bwd_microstep: 219.32 | bwd_inner_microstep: 219.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 165.39 | bwd_microstep: 231.76 | bwd_inner_microstep: 231.63 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 165.62 | bwd_microstep: 243.43 | bwd_inner_microstep: 243.29 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 162.31 | bwd_microstep: 219.85 | bwd_inner_microstep: 219.74 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.07 | bwd_microstep: 233.02 | bwd_inner_microstep: 232.73 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 164.22 | bwd_microstep: 221.50 | bwd_inner_microstep: 221.37 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.93 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.23 | bwd_microstep: 257.68 | bwd_inner_microstep: 244.15 | bwd_allreduce_microstep: 13.37 | step_microstep: 21.15 [Rank 0] time (ms) | fwd: 1311.02 | bwd: 1841.11 | bwd_inner: 1826.72 | bwd_allreduce: 13.81 | step: 21.66 7%|▋ | 208/2865 [1:21:32<3:28:40, 4.71s/it] {'loss': 1.111, 'learning_rate': 3.981008739241685e-05, 'epoch': 0.22} 7%|▋ | 208/2865 [1:21:32<3:28:40, 4.71s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 163.71 | bwd_microstep: 223.49 | bwd_inner_microstep: 223.14 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 162.27 | bwd_microstep: 225.18 | bwd_inner_microstep: 224.79 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 166.19 | bwd_microstep: 230.79 | bwd_inner_microstep: 230.59 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 172.08 | bwd_microstep: 232.03 | bwd_inner_microstep: 231.85 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 170.35 | bwd_microstep: 227.22 | bwd_inner_microstep: 226.88 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 165.41 | bwd_microstep: 227.37 | bwd_inner_microstep: 227.24 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 163.02 | bwd_microstep: 230.70 | bwd_inner_microstep: 230.56 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.97 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 166.11 | bwd_microstep: 262.63 | bwd_inner_microstep: 251.66 | bwd_allreduce_microstep: 10.74 | step_microstep: 23.54 [Rank 0] time (ms) | fwd: 1329.02 | bwd: 1859.56 | bwd_inner: 1847.11 | bwd_allreduce: 11.42 | step: 24.38 7%|▋ | 209/2865 [1:21:36<3:10:49, 4.31s/it] {'loss': 0.9456, 'learning_rate': 3.980696634977873e-05, 'epoch': 0.22} 7%|▋ | 209/2865 [1:21:36<3:10:49, 4.31s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 946 [Rank 0] time (ms) | fwd_microstep: 170.28 | bwd_microstep: 223.10 | bwd_inner_microstep: 222.94 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 942 [Rank 0] time (ms) | fwd_microstep: 169.22 | bwd_microstep: 318.13 | bwd_inner_microstep: 317.99 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 198.90 | bwd_microstep: 265.08 | bwd_inner_microstep: 264.70 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.27 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 186.91 | bwd_microstep: 324.84 | bwd_inner_microstep: 324.34 | bwd_allreduce_microstep: 0.23 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 182.82 | bwd_microstep: 275.27 | bwd_inner_microstep: 275.05 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 198.49 | bwd_microstep: 275.72 | bwd_inner_microstep: 275.45 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 181.47 | bwd_microstep: 278.70 | bwd_inner_microstep: 278.42 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.00 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 176.40 | bwd_microstep: 312.62 | bwd_inner_microstep: 301.40 | bwd_allreduce_microstep: 11.10 | step_microstep: 22.08 [Rank 0] time (ms) | fwd: 1464.34 | bwd: 2273.62 | bwd_inner: 2260.72 | bwd_allreduce: 11.93 | step: 22.93 7%|▋ | 210/2865 [1:21:40<3:05:50, 4.20s/it] {'loss': 1.1261, 'learning_rate': 3.9803819994318375e-05, 'epoch': 0.22} 7%|▋ | 210/2865 [1:21:40<3:05:50, 4.20s/it][INFO|trainer.py:3242] 2026-05-14 21:42:22,799 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 21:42:22,800 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 21:42:22,800 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-210 [INFO|configuration_utils.py:473] 2026-05-14 21:47:29,386 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-210/config.json [INFO|configuration_utils.py:594] 2026-05-14 21:47:29,395 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-210/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 21:47:34,532 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-210/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 21:47:34,560 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-210/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 21:47:34,569 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-210/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 21:47:34,571 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-210/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 21:47:43,347 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-135] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 303032.37 | bwd_microstep: 243.80 | bwd_inner_microstep: 243.59 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 192.59 | bwd_microstep: 275.34 | bwd_inner_microstep: 275.20 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 170.93 | bwd_microstep: 231.44 | bwd_inner_microstep: 231.12 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 170.20 | bwd_microstep: 272.15 | bwd_inner_microstep: 271.97 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 167.42 | bwd_microstep: 227.72 | bwd_inner_microstep: 227.42 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 165.65 | bwd_microstep: 249.50 | bwd_inner_microstep: 249.36 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 146.41 | bwd_microstep: 197.78 | bwd_inner_microstep: 197.47 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 161.14 | bwd_microstep: 258.88 | bwd_inner_microstep: 248.57 | bwd_allreduce_microstep: 10.15 | step_microstep: 21.31 [Rank 0] time (ms) | fwd: 304196.00 | bwd: 1956.71 | bwd_inner: 1945.02 | bwd_allreduce: 10.87 | step: 21.93 7%|▋ | 211/2865 [1:27:04<73:57:42, 100.33s/it] {'loss': 1.0561, 'learning_rate': 3.980064833005674e-05, 'epoch': 0.22} 7%|▋ | 211/2865 [1:27:04<73:57:42, 100.33s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 948 [Rank 0] time (ms) | fwd_microstep: 165.72 | bwd_microstep: 228.54 | bwd_inner_microstep: 228.21 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 162.80 | bwd_microstep: 224.86 | bwd_inner_microstep: 224.60 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 162.29 | bwd_microstep: 223.11 | bwd_inner_microstep: 222.85 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 162.38 | bwd_microstep: 228.50 | bwd_inner_microstep: 228.36 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.42 | bwd_microstep: 234.41 | bwd_inner_microstep: 234.04 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.55 | bwd_microstep: 231.25 | bwd_inner_microstep: 231.11 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 163.17 | bwd_microstep: 223.52 | bwd_inner_microstep: 223.27 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.97 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 167.03 | bwd_microstep: 258.85 | bwd_inner_microstep: 249.09 | bwd_allreduce_microstep: 9.68 | step_microstep: 22.35 [Rank 0] time (ms) | fwd: 1304.24 | bwd: 1853.17 | bwd_inner: 1841.88 | bwd_allreduce: 10.41 | step: 23.09 7%|▋ | 212/2865 [1:27:08<52:29:26, 71.23s/it] {'loss': 1.1362, 'learning_rate': 3.979745136104714e-05, 'epoch': 0.22} 7%|▋ | 212/2865 [1:27:08<52:29:26, 71.23s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 173.59 | bwd_microstep: 221.83 | bwd_inner_microstep: 221.64 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 162.80 | bwd_microstep: 224.70 | bwd_inner_microstep: 224.48 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 174.57 | bwd_microstep: 233.22 | bwd_inner_microstep: 233.09 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 162.26 | bwd_microstep: 229.04 | bwd_inner_microstep: 228.69 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 165.44 | bwd_microstep: 239.50 | bwd_inner_microstep: 239.11 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 164.30 | bwd_microstep: 257.62 | bwd_inner_microstep: 257.32 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 165.31 | bwd_microstep: 227.40 | bwd_inner_microstep: 227.19 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.13 | bwd_microstep: 263.37 | bwd_inner_microstep: 250.94 | bwd_allreduce_microstep: 12.32 | step_microstep: 21.20 [Rank 0] time (ms) | fwd: 1330.26 | bwd: 1896.81 | bwd_inner: 1882.86 | bwd_allreduce: 13.05 | step: 21.97 7%|▋ | 213/2865 [1:27:11<37:28:56, 50.88s/it] {'loss': 1.0523, 'learning_rate': 3.979422909137522e-05, 'epoch': 0.22} 7%|▋ | 213/2865 [1:27:11<37:28:56, 50.88s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 161.63 | bwd_microstep: 221.49 | bwd_inner_microstep: 221.18 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 161.99 | bwd_microstep: 221.91 | bwd_inner_microstep: 221.68 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 163.73 | bwd_microstep: 237.17 | bwd_inner_microstep: 236.79 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 163.95 | bwd_microstep: 245.48 | bwd_inner_microstep: 245.19 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 172.00 | bwd_microstep: 220.91 | bwd_inner_microstep: 220.71 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 162.82 | bwd_microstep: 240.27 | bwd_inner_microstep: 240.11 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 175.30 | bwd_microstep: 265.24 | bwd_inner_microstep: 264.90 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.52 | bwd_microstep: 259.62 | bwd_inner_microstep: 249.83 | bwd_allreduce_microstep: 9.71 | step_microstep: 21.33 [Rank 0] time (ms) | fwd: 1323.81 | bwd: 1912.24 | bwd_inner: 1900.77 | bwd_allreduce: 10.52 | step: 22.11 7%|▋ | 214/2865 [1:27:14<26:58:57, 36.64s/it] {'loss': 1.0797, 'learning_rate': 3.9790981525158964e-05, 'epoch': 0.22} 7%|▋ | 214/2865 [1:27:14<26:58:57, 36.64s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 940 [Rank 0] time (ms) | fwd_microstep: 164.25 | bwd_microstep: 217.53 | bwd_inner_microstep: 217.14 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 167.29 | bwd_microstep: 221.84 | bwd_inner_microstep: 221.46 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 169.87 | bwd_microstep: 226.17 | bwd_inner_microstep: 225.83 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 166.46 | bwd_microstep: 218.71 | bwd_inner_microstep: 218.50 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 177.18 | bwd_microstep: 220.16 | bwd_inner_microstep: 219.94 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.98 | bwd_microstep: 220.03 | bwd_inner_microstep: 219.48 | bwd_allreduce_microstep: 0.21 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 162.85 | bwd_microstep: 222.77 | bwd_inner_microstep: 222.57 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 167.71 | bwd_microstep: 258.50 | bwd_inner_microstep: 248.54 | bwd_allreduce_microstep: 9.76 | step_microstep: 21.33 [Rank 0] time (ms) | fwd: 1337.47 | bwd: 1805.87 | bwd_inner: 1794.01 | bwd_allreduce: 10.66 | step: 22.22 8%|▊ | 215/2865 [1:27:18<19:36:55, 26.65s/it] {'loss': 1.1345, 'learning_rate': 3.978770866654869e-05, 'epoch': 0.22} 8%|▊ | 215/2865 [1:27:18<19:36:55, 26.65s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 946 [Rank 0] time (ms) | fwd_microstep: 165.57 | bwd_microstep: 221.53 | bwd_inner_microstep: 221.43 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 164.22 | bwd_microstep: 226.94 | bwd_inner_microstep: 226.58 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 163.24 | bwd_microstep: 233.77 | bwd_inner_microstep: 233.58 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 164.25 | bwd_microstep: 233.90 | bwd_inner_microstep: 233.73 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 172.76 | bwd_microstep: 230.04 | bwd_inner_microstep: 229.87 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 160.75 | bwd_microstep: 238.00 | bwd_inner_microstep: 237.67 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 166.06 | bwd_microstep: 237.93 | bwd_inner_microstep: 237.80 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 168.39 | bwd_microstep: 259.35 | bwd_inner_microstep: 249.55 | bwd_allreduce_microstep: 9.73 | step_microstep: 21.14 [Rank 0] time (ms) | fwd: 1325.10 | bwd: 1881.58 | bwd_inner: 1870.48 | bwd_allreduce: 10.38 | step: 21.77 8%|▊ | 216/2865 [1:27:21<14:28:18, 19.67s/it] {'loss': 1.1091, 'learning_rate': 3.9784410519727015e-05, 'epoch': 0.23} 8%|▊ | 216/2865 [1:27:21<14:28:18, 19.67s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 170.56 | bwd_microstep: 225.11 | bwd_inner_microstep: 224.93 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 179.83 | bwd_microstep: 227.44 | bwd_inner_microstep: 227.24 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 164.00 | bwd_microstep: 227.75 | bwd_inner_microstep: 227.59 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 163.54 | bwd_microstep: 232.32 | bwd_inner_microstep: 231.93 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.01 | bwd_microstep: 226.83 | bwd_inner_microstep: 226.64 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 162.47 | bwd_microstep: 226.55 | bwd_inner_microstep: 226.18 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 163.92 | bwd_microstep: 228.42 | bwd_inner_microstep: 228.02 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.93 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 162.70 | bwd_microstep: 260.53 | bwd_inner_microstep: 247.52 | bwd_allreduce_microstep: 12.84 | step_microstep: 23.14 [Rank 0] time (ms) | fwd: 1328.92 | bwd: 1855.16 | bwd_inner: 1840.43 | bwd_allreduce: 13.65 | step: 24.09 8%|▊ | 217/2865 [1:27:24<10:52:08, 14.78s/it] {'loss': 1.1692, 'learning_rate': 3.978108708890891e-05, 'epoch': 0.23} 8%|▊ | 217/2865 [1:27:24<10:52:08, 14.78s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 163.12 | bwd_microstep: 221.40 | bwd_inner_microstep: 221.27 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 188.31 | bwd_microstep: 224.84 | bwd_inner_microstep: 224.67 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 162.28 | bwd_microstep: 239.02 | bwd_inner_microstep: 238.66 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 163.76 | bwd_microstep: 246.57 | bwd_inner_microstep: 246.37 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 185.47 | bwd_microstep: 225.65 | bwd_inner_microstep: 225.49 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.94 | bwd_microstep: 254.62 | bwd_inner_microstep: 254.47 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 168.97 | bwd_microstep: 227.73 | bwd_inner_microstep: 227.58 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.97 | bwd_microstep: 269.77 | bwd_inner_microstep: 259.89 | bwd_allreduce_microstep: 9.72 | step_microstep: 21.10 [Rank 0] time (ms) | fwd: 1356.69 | bwd: 1909.73 | bwd_inner: 1898.70 | bwd_allreduce: 10.27 | step: 21.82 8%|▊ | 218/2865 [1:27:28<8:21:57, 11.38s/it] {'loss': 1.1035, 'learning_rate': 3.9777738378341626e-05, 'epoch': 0.23} 8%|▊ | 218/2865 [1:27:28<8:21:57, 11.38s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 163.54 | bwd_microstep: 222.12 | bwd_inner_microstep: 222.00 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 938 [Rank 0] time (ms) | fwd_microstep: 164.38 | bwd_microstep: 240.48 | bwd_inner_microstep: 240.28 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 164.87 | bwd_microstep: 221.79 | bwd_inner_microstep: 221.59 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.59 | bwd_microstep: 229.42 | bwd_inner_microstep: 229.22 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 164.47 | bwd_microstep: 221.31 | bwd_inner_microstep: 221.08 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 164.48 | bwd_microstep: 230.19 | bwd_inner_microstep: 230.01 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.97 | bwd_microstep: 224.61 | bwd_inner_microstep: 224.37 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 164.47 | bwd_microstep: 261.42 | bwd_inner_microstep: 251.53 | bwd_allreduce_microstep: 9.79 | step_microstep: 21.39 [Rank 0] time (ms) | fwd: 1309.66 | bwd: 1851.47 | bwd_inner: 1840.35 | bwd_allreduce: 10.38 | step: 22.13 8%|▊ | 219/2865 [1:27:31<6:35:23, 8.97s/it] {'loss': 0.9682, 'learning_rate': 3.977436439230474e-05, 'epoch': 0.23} 8%|▊ | 219/2865 [1:27:31<6:35:23, 8.97s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 163.46 | bwd_microstep: 255.31 | bwd_inner_microstep: 255.12 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 162.46 | bwd_microstep: 243.83 | bwd_inner_microstep: 243.64 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 166.38 | bwd_microstep: 224.63 | bwd_inner_microstep: 224.49 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 166.39 | bwd_microstep: 239.03 | bwd_inner_microstep: 238.88 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 167.52 | bwd_microstep: 247.36 | bwd_inner_microstep: 247.04 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.32 | bwd_microstep: 224.49 | bwd_inner_microstep: 224.34 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.19 | bwd_microstep: 228.27 | bwd_inner_microstep: 227.97 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.88 | bwd_microstep: 271.32 | bwd_inner_microstep: 255.89 | bwd_allreduce_microstep: 15.24 | step_microstep: 21.35 [Rank 0] time (ms) | fwd: 1313.48 | bwd: 1934.37 | bwd_inner: 1917.67 | bwd_allreduce: 15.87 | step: 22.11 8%|▊ | 220/2865 [1:27:35<5:21:59, 7.30s/it] {'loss': 1.0829, 'learning_rate': 3.977096513511012e-05, 'epoch': 0.23} 8%|▊ | 220/2865 [1:27:35<5:21:59, 7.30s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 958 [Rank 0] time (ms) | fwd_microstep: 169.99 | bwd_microstep: 221.65 | bwd_inner_microstep: 221.54 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 164.11 | bwd_microstep: 239.61 | bwd_inner_microstep: 239.36 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 166.93 | bwd_microstep: 230.46 | bwd_inner_microstep: 230.15 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 173.07 | bwd_microstep: 234.41 | bwd_inner_microstep: 234.14 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 165.51 | bwd_microstep: 244.45 | bwd_inner_microstep: 244.08 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 161.58 | bwd_microstep: 258.16 | bwd_inner_microstep: 257.96 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 164.67 | bwd_microstep: 284.95 | bwd_inner_microstep: 284.71 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 165.60 | bwd_microstep: 297.34 | bwd_inner_microstep: 286.99 | bwd_allreduce_microstep: 10.25 | step_microstep: 21.32 [Rank 0] time (ms) | fwd: 1331.33 | bwd: 2011.17 | bwd_inner: 1999.34 | bwd_allreduce: 11.00 | step: 22.04 8%|▊ | 221/2865 [1:27:38<4:31:49, 6.17s/it] {'loss': 1.0018, 'learning_rate': 3.976754061110193e-05, 'epoch': 0.23} 8%|▊ | 221/2865 [1:27:38<4:31:49, 6.17s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 163.61 | bwd_microstep: 247.90 | bwd_inner_microstep: 247.78 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 162.90 | bwd_microstep: 231.98 | bwd_inner_microstep: 231.76 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.81 | bwd_microstep: 232.18 | bwd_inner_microstep: 231.96 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 165.83 | bwd_microstep: 243.37 | bwd_inner_microstep: 242.95 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 162.72 | bwd_microstep: 227.25 | bwd_inner_microstep: 226.91 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 166.28 | bwd_microstep: 220.67 | bwd_inner_microstep: 220.45 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 162.11 | bwd_microstep: 232.31 | bwd_inner_microstep: 232.18 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 147.66 | bwd_microstep: 239.93 | bwd_inner_microstep: 229.04 | bwd_allreduce_microstep: 10.74 | step_microstep: 21.23 [Rank 0] time (ms) | fwd: 1291.80 | bwd: 1875.72 | bwd_inner: 1863.43 | bwd_allreduce: 11.42 | step: 21.85 8%|▊ | 222/2865 [1:27:42<3:54:21, 5.32s/it] {'loss': 1.0959, 'learning_rate': 3.976409082465665e-05, 'epoch': 0.23} 8%|▊ | 222/2865 [1:27:42<3:54:21, 5.32s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 942 [Rank 0] time (ms) | fwd_microstep: 164.38 | bwd_microstep: 217.91 | bwd_inner_microstep: 217.78 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 171.31 | bwd_microstep: 221.36 | bwd_inner_microstep: 221.22 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 181.78 | bwd_microstep: 252.61 | bwd_inner_microstep: 252.42 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 161.91 | bwd_microstep: 237.54 | bwd_inner_microstep: 237.22 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 165.08 | bwd_microstep: 267.54 | bwd_inner_microstep: 267.39 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 161.48 | bwd_microstep: 244.71 | bwd_inner_microstep: 244.53 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 162.63 | bwd_microstep: 245.18 | bwd_inner_microstep: 244.78 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.99 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.15 | bwd_microstep: 275.29 | bwd_inner_microstep: 264.51 | bwd_allreduce_microstep: 10.63 | step_microstep: 21.23 [Rank 0] time (ms) | fwd: 1330.60 | bwd: 1962.26 | bwd_inner: 1950.22 | bwd_allreduce: 11.25 | step: 21.90 8%|▊ | 223/2865 [1:27:45<3:29:51, 4.77s/it] {'loss': 0.9708, 'learning_rate': 3.9760615780183e-05, 'epoch': 0.23} 8%|▊ | 223/2865 [1:27:45<3:29:51, 4.77s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 163.07 | bwd_microstep: 233.82 | bwd_inner_microstep: 233.69 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 164.95 | bwd_microstep: 229.08 | bwd_inner_microstep: 228.92 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 162.17 | bwd_microstep: 254.71 | bwd_inner_microstep: 254.45 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 175.08 | bwd_microstep: 295.17 | bwd_inner_microstep: 294.74 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 175.73 | bwd_microstep: 286.34 | bwd_inner_microstep: 286.15 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 194.33 | bwd_microstep: 296.22 | bwd_inner_microstep: 295.99 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 178.86 | bwd_microstep: 275.49 | bwd_inner_microstep: 275.30 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.99 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 174.50 | bwd_microstep: 350.84 | bwd_inner_microstep: 339.93 | bwd_allreduce_microstep: 10.75 | step_microstep: 21.51 [Rank 0] time (ms) | fwd: 1388.54 | bwd: 2221.79 | bwd_inner: 2209.55 | bwd_allreduce: 11.39 | step: 22.36 8%|▊ | 224/2865 [1:27:49<3:17:10, 4.48s/it] {'loss': 1.0201, 'learning_rate': 3.9757115482122006e-05, 'epoch': 0.23} 8%|▊ | 224/2865 [1:27:49<3:17:10, 4.48s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 180.06 | bwd_microstep: 272.16 | bwd_inner_microstep: 271.77 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 174.03 | bwd_microstep: 282.10 | bwd_inner_microstep: 281.76 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 176.44 | bwd_microstep: 284.06 | bwd_inner_microstep: 283.67 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 165.03 | bwd_microstep: 255.19 | bwd_inner_microstep: 254.80 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 166.48 | bwd_microstep: 279.64 | bwd_inner_microstep: 279.40 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 174.16 | bwd_microstep: 298.52 | bwd_inner_microstep: 298.31 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 165.74 | bwd_microstep: 278.16 | bwd_inner_microstep: 278.00 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.24 | bwd_microstep: 263.69 | bwd_inner_microstep: 253.51 | bwd_allreduce_microstep: 10.10 | step_microstep: 21.08 [Rank 0] time (ms) | fwd: 1362.03 | bwd: 2213.70 | bwd_inner: 2201.63 | bwd_allreduce: 10.97 | step: 22.07 8%|▊ | 225/2865 [1:27:53<3:07:47, 4.27s/it] {'loss': 1.0763, 'learning_rate': 3.975358993494697e-05, 'epoch': 0.24} 8%|▊ | 225/2865 [1:27:53<3:07:47, 4.27s/it][INFO|trainer.py:3242] 2026-05-14 21:48:35,774 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 21:48:35,774 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 21:48:35,774 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-225 [INFO|configuration_utils.py:473] 2026-05-14 21:53:42,123 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-225/config.json [INFO|configuration_utils.py:594] 2026-05-14 21:53:42,130 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-225/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 21:53:47,076 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-225/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 21:53:47,090 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-225/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 21:53:47,099 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-225/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 21:53:47,109 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-225/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 21:53:56,014 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-150] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 303122.13 | bwd_microstep: 227.24 | bwd_inner_microstep: 227.01 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.42 | bwd_microstep: 234.62 | bwd_inner_microstep: 234.30 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.83 | bwd_microstep: 234.74 | bwd_inner_microstep: 234.61 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 161.54 | bwd_microstep: 225.70 | bwd_inner_microstep: 225.58 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 160.78 | bwd_microstep: 234.61 | bwd_inner_microstep: 234.22 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.10 | bwd_microstep: 229.92 | bwd_inner_microstep: 229.74 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 165.40 | bwd_microstep: 224.99 | bwd_inner_microstep: 224.80 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.01 | optimizer_gradients: 3.02 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 164.12 | bwd_microstep: 267.82 | bwd_inner_microstep: 257.13 | bwd_allreduce_microstep: 10.60 | step_microstep: 21.70 [Rank 0] time (ms) | fwd: 304246.28 | bwd: 1879.78 | bwd_inner: 1867.65 | bwd_allreduce: 11.32 | step: 22.29 8%|▊ | 226/2865 [1:33:17<73:28:44, 100.24s/it] {'loss': 1.1545, 'learning_rate': 3.975003914316346e-05, 'epoch': 0.24} 8%|▊ | 226/2865 [1:33:17<73:28:44, 100.24s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 171.60 | bwd_microstep: 242.28 | bwd_inner_microstep: 241.88 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 163.11 | bwd_microstep: 220.03 | bwd_inner_microstep: 219.72 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 167.77 | bwd_microstep: 223.67 | bwd_inner_microstep: 223.53 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 185.65 | bwd_microstep: 238.63 | bwd_inner_microstep: 238.44 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 165.29 | bwd_microstep: 221.02 | bwd_inner_microstep: 220.65 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 163.75 | bwd_microstep: 223.50 | bwd_inner_microstep: 223.37 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 167.19 | bwd_microstep: 228.58 | bwd_inner_microstep: 228.26 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 166.30 | bwd_microstep: 264.72 | bwd_inner_microstep: 254.69 | bwd_allreduce_microstep: 9.88 | step_microstep: 21.24 [Rank 0] time (ms) | fwd: 1350.56 | bwd: 1862.55 | bwd_inner: 1850.97 | bwd_allreduce: 10.60 | step: 21.95 8%|▊ | 227/2865 [1:33:20<52:09:41, 71.18s/it] {'loss': 0.9185, 'learning_rate': 3.974646311130929e-05, 'epoch': 0.24} 8%|▊ | 227/2865 [1:33:20<52:09:41, 71.18s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 943 [Rank 0] time (ms) | fwd_microstep: 171.94 | bwd_microstep: 228.47 | bwd_inner_microstep: 228.13 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 164.10 | bwd_microstep: 223.32 | bwd_inner_microstep: 223.14 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.83 | bwd_microstep: 223.97 | bwd_inner_microstep: 223.59 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 165.88 | bwd_microstep: 227.50 | bwd_inner_microstep: 227.18 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 163.57 | bwd_microstep: 225.39 | bwd_inner_microstep: 225.18 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 163.83 | bwd_microstep: 226.47 | bwd_inner_microstep: 226.29 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 171.08 | bwd_microstep: 229.39 | bwd_inner_microstep: 229.25 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 170.79 | bwd_microstep: 258.40 | bwd_inner_microstep: 246.87 | bwd_allreduce_microstep: 11.41 | step_microstep: 21.36 [Rank 0] time (ms) | fwd: 1332.88 | bwd: 1843.07 | bwd_inner: 1829.93 | bwd_allreduce: 12.19 | step: 22.19 8%|▊ | 228/2865 [1:33:24<37:14:20, 50.84s/it] {'loss': 1.0277, 'learning_rate': 3.974286184395455e-05, 'epoch': 0.24} 8%|▊ | 228/2865 [1:33:24<37:14:20, 50.84s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 941 [Rank 0] time (ms) | fwd_microstep: 165.51 | bwd_microstep: 222.18 | bwd_inner_microstep: 222.00 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 163.77 | bwd_microstep: 228.17 | bwd_inner_microstep: 228.00 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 162.06 | bwd_microstep: 230.40 | bwd_inner_microstep: 230.25 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 164.61 | bwd_microstep: 231.52 | bwd_inner_microstep: 231.28 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 162.45 | bwd_microstep: 231.99 | bwd_inner_microstep: 231.91 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 162.74 | bwd_microstep: 227.88 | bwd_inner_microstep: 227.61 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 174.72 | bwd_microstep: 225.58 | bwd_inner_microstep: 225.47 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.60 | bwd_microstep: 252.53 | bwd_inner_microstep: 242.48 | bwd_allreduce_microstep: 9.93 | step_microstep: 21.22 [Rank 0] time (ms) | fwd: 1318.37 | bwd: 1850.36 | bwd_inner: 1839.20 | bwd_allreduce: 10.51 | step: 21.90 8%|▊ | 229/2865 [1:33:27<26:47:18, 36.59s/it] {'loss': 1.2275, 'learning_rate': 3.973923534570157e-05, 'epoch': 0.24} 8%|▊ | 229/2865 [1:33:27<26:47:18, 36.59s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 168.76 | bwd_microstep: 221.13 | bwd_inner_microstep: 221.00 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 162.78 | bwd_microstep: 219.62 | bwd_inner_microstep: 219.50 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 165.55 | bwd_microstep: 232.39 | bwd_inner_microstep: 232.31 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 161.55 | bwd_microstep: 232.81 | bwd_inner_microstep: 232.68 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 162.03 | bwd_microstep: 230.04 | bwd_inner_microstep: 229.95 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 162.01 | bwd_microstep: 226.05 | bwd_inner_microstep: 225.96 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 162.09 | bwd_microstep: 224.95 | bwd_inner_microstep: 224.87 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.23 | bwd_microstep: 263.80 | bwd_inner_microstep: 253.88 | bwd_allreduce_microstep: 9.85 | step_microstep: 21.12 [Rank 0] time (ms) | fwd: 1306.89 | bwd: 1850.86 | bwd_inner: 1840.22 | bwd_allreduce: 10.26 | step: 21.64 8%|▊ | 230/2865 [1:33:30<19:28:27, 26.61s/it] {'loss': 1.2053, 'learning_rate': 3.973558362118493e-05, 'epoch': 0.24} 8%|▊ | 230/2865 [1:33:30<19:28:27, 26.61s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 175.88 | bwd_microstep: 223.72 | bwd_inner_microstep: 223.36 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.76 | bwd_microstep: 224.31 | bwd_inner_microstep: 223.94 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 175.61 | bwd_microstep: 224.98 | bwd_inner_microstep: 224.61 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 170.41 | bwd_microstep: 263.95 | bwd_inner_microstep: 263.62 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 166.97 | bwd_microstep: 234.41 | bwd_inner_microstep: 234.28 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 164.14 | bwd_microstep: 224.11 | bwd_inner_microstep: 223.71 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 165.25 | bwd_microstep: 264.12 | bwd_inner_microstep: 263.94 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.91 | bwd_microstep: 251.76 | bwd_inner_microstep: 241.98 | bwd_allreduce_microstep: 9.71 | step_microstep: 21.04 [Rank 0] time (ms) | fwd: 1342.82 | bwd: 1911.53 | bwd_inner: 1899.93 | bwd_allreduce: 10.58 | step: 21.72 8%|▊ | 231/2865 [1:33:34<14:22:57, 19.66s/it] {'loss': 1.0772, 'learning_rate': 3.973190667507144e-05, 'epoch': 0.24} 8%|▊ | 231/2865 [1:33:34<14:22:57, 19.66s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.85 | bwd_microstep: 218.84 | bwd_inner_microstep: 218.75 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 162.90 | bwd_microstep: 223.04 | bwd_inner_microstep: 222.96 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.97 | bwd_microstep: 219.85 | bwd_inner_microstep: 219.77 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 163.78 | bwd_microstep: 221.85 | bwd_inner_microstep: 221.77 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 166.17 | bwd_microstep: 223.59 | bwd_inner_microstep: 223.51 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 165.64 | bwd_microstep: 231.87 | bwd_inner_microstep: 231.77 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.98 | bwd_microstep: 223.24 | bwd_inner_microstep: 223.17 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 162.60 | bwd_microstep: 252.20 | bwd_inner_microstep: 242.13 | bwd_allreduce_microstep: 9.98 | step_microstep: 21.02 [Rank 0] time (ms) | fwd: 1305.78 | bwd: 1814.56 | bwd_inner: 1803.86 | bwd_allreduce: 10.37 | step: 21.53 8%|▊ | 232/2865 [1:33:37<10:47:01, 14.74s/it] {'loss': 0.9095, 'learning_rate': 3.972820451206015e-05, 'epoch': 0.24} 8%|▊ | 232/2865 [1:33:37<10:47:01, 14.74s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 161.94 | bwd_microstep: 218.30 | bwd_inner_microstep: 218.21 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.39 | bwd_microstep: 224.01 | bwd_inner_microstep: 223.90 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 167.78 | bwd_microstep: 227.35 | bwd_inner_microstep: 227.16 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 162.55 | bwd_microstep: 232.99 | bwd_inner_microstep: 232.63 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 171.98 | bwd_microstep: 231.24 | bwd_inner_microstep: 231.10 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 163.41 | bwd_microstep: 225.14 | bwd_inner_microstep: 224.84 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 164.93 | bwd_microstep: 229.25 | bwd_inner_microstep: 228.89 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 150.08 | bwd_microstep: 231.63 | bwd_inner_microstep: 219.69 | bwd_allreduce_microstep: 11.84 | step_microstep: 21.34 [Rank 0] time (ms) | fwd: 1303.94 | bwd: 1820.06 | bwd_inner: 1806.69 | bwd_allreduce: 12.55 | step: 22.12 8%|▊ | 233/2865 [1:33:40<8:16:10, 11.31s/it] {'loss': 1.1663, 'learning_rate': 3.972447713688233e-05, 'epoch': 0.24} 8%|▊ | 233/2865 [1:33:40<8:16:10, 11.31s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 166.89 | bwd_microstep: 221.99 | bwd_inner_microstep: 221.61 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 168.09 | bwd_microstep: 233.46 | bwd_inner_microstep: 233.06 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 173.87 | bwd_microstep: 258.34 | bwd_inner_microstep: 258.02 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.76 | bwd_microstep: 229.31 | bwd_inner_microstep: 228.90 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 169.35 | bwd_microstep: 227.85 | bwd_inner_microstep: 227.47 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 161.49 | bwd_microstep: 224.77 | bwd_inner_microstep: 224.62 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 165.82 | bwd_microstep: 255.46 | bwd_inner_microstep: 255.06 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 169.74 | bwd_microstep: 273.25 | bwd_inner_microstep: 263.19 | bwd_allreduce_microstep: 9.94 | step_microstep: 20.97 [Rank 0] time (ms) | fwd: 1337.88 | bwd: 1924.62 | bwd_inner: 1912.43 | bwd_allreduce: 11.03 | step: 22.05 8%|▊ | 234/2865 [1:33:44<6:32:32, 8.95s/it] {'loss': 1.1914, 'learning_rate': 3.972072455430149e-05, 'epoch': 0.24} 8%|▊ | 234/2865 [1:33:44<6:32:32, 8.95s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 169.81 | bwd_microstep: 233.70 | bwd_inner_microstep: 233.56 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 165.46 | bwd_microstep: 226.56 | bwd_inner_microstep: 226.16 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.98 | bwd_microstep: 227.47 | bwd_inner_microstep: 227.19 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 162.67 | bwd_microstep: 231.64 | bwd_inner_microstep: 231.33 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 162.44 | bwd_microstep: 232.11 | bwd_inner_microstep: 231.84 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 165.60 | bwd_microstep: 237.01 | bwd_inner_microstep: 236.87 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 909 [Rank 0] time (ms) | fwd_microstep: 164.25 | bwd_microstep: 213.76 | bwd_inner_microstep: 213.43 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 164.91 | bwd_microstep: 265.99 | bwd_inner_microstep: 254.00 | bwd_allreduce_microstep: 11.80 | step_microstep: 24.44 [Rank 0] time (ms) | fwd: 1316.95 | bwd: 1868.39 | bwd_inner: 1854.85 | bwd_allreduce: 12.52 | step: 25.17 8%|▊ | 235/2865 [1:33:47<5:19:05, 7.28s/it] {'loss': 0.955, 'learning_rate': 3.971694676911332e-05, 'epoch': 0.25} 8%|▊ | 235/2865 [1:33:47<5:19:05, 7.28s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 167.58 | bwd_microstep: 227.04 | bwd_inner_microstep: 226.73 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 177.49 | bwd_microstep: 221.36 | bwd_inner_microstep: 221.03 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 177.69 | bwd_microstep: 246.01 | bwd_inner_microstep: 245.73 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 167.52 | bwd_microstep: 241.61 | bwd_inner_microstep: 241.39 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 164.30 | bwd_microstep: 227.51 | bwd_inner_microstep: 227.29 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.41 | bwd_microstep: 249.48 | bwd_inner_microstep: 249.34 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 170.99 | bwd_microstep: 225.40 | bwd_inner_microstep: 225.19 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 150.04 | bwd_microstep: 233.86 | bwd_inner_microstep: 223.56 | bwd_allreduce_microstep: 10.15 | step_microstep: 21.00 [Rank 0] time (ms) | fwd: 1337.82 | bwd: 1872.41 | bwd_inner: 1860.68 | bwd_allreduce: 10.84 | step: 21.71 8%|▊ | 236/2865 [1:33:50<4:28:01, 6.12s/it] {'loss': 0.9635, 'learning_rate': 3.9713143786145754e-05, 'epoch': 0.25} 8%|▊ | 236/2865 [1:33:50<4:28:01, 6.12s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 163.38 | bwd_microstep: 232.77 | bwd_inner_microstep: 232.63 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 162.63 | bwd_microstep: 243.30 | bwd_inner_microstep: 243.01 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 167.19 | bwd_microstep: 231.95 | bwd_inner_microstep: 231.74 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 162.95 | bwd_microstep: 234.14 | bwd_inner_microstep: 234.00 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 167.39 | bwd_microstep: 231.86 | bwd_inner_microstep: 231.68 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 163.60 | bwd_microstep: 254.45 | bwd_inner_microstep: 254.29 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.32 | bwd_microstep: 248.45 | bwd_inner_microstep: 248.27 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.25 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 161.78 | bwd_microstep: 268.83 | bwd_inner_microstep: 258.12 | bwd_allreduce_microstep: 10.63 | step_microstep: 21.88 [Rank 0] time (ms) | fwd: 1312.12 | bwd: 1945.87 | bwd_inner: 1934.04 | bwd_allreduce: 11.18 | step: 22.47 8%|▊ | 237/2865 [1:33:54<3:52:42, 5.31s/it] {'loss': 1.2736, 'learning_rate': 3.97093156102589e-05, 'epoch': 0.25} 8%|▊ | 237/2865 [1:33:54<3:52:42, 5.31s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 165.23 | bwd_microstep: 221.30 | bwd_inner_microstep: 221.10 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 172.50 | bwd_microstep: 233.44 | bwd_inner_microstep: 233.30 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 163.66 | bwd_microstep: 230.27 | bwd_inner_microstep: 230.13 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 166.19 | bwd_microstep: 231.31 | bwd_inner_microstep: 231.06 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 162.73 | bwd_microstep: 238.39 | bwd_inner_microstep: 238.18 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 162.13 | bwd_microstep: 221.78 | bwd_inner_microstep: 221.45 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 166.43 | bwd_microstep: 227.94 | bwd_inner_microstep: 227.84 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.57 | bwd_microstep: 262.77 | bwd_inner_microstep: 253.05 | bwd_allreduce_microstep: 9.63 | step_microstep: 80.97 [Rank 0] time (ms) | fwd: 1321.31 | bwd: 1867.29 | bwd_inner: 1856.40 | bwd_allreduce: 10.20 | step: 81.60 8%|▊ | 238/2865 [1:33:57<3:27:54, 4.75s/it] {'loss': 0.8736, 'learning_rate': 3.9705462246345076e-05, 'epoch': 0.25} 8%|▊ | 238/2865 [1:33:57<3:27:54, 4.75s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 940 [Rank 0] time (ms) | fwd_microstep: 162.75 | bwd_microstep: 221.48 | bwd_inner_microstep: 221.37 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 162.22 | bwd_microstep: 217.86 | bwd_inner_microstep: 217.80 | bwd_allreduce_microstep: 0.03 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.12 | bwd_microstep: 224.78 | bwd_inner_microstep: 224.69 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 161.59 | bwd_microstep: 220.66 | bwd_inner_microstep: 220.58 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 162.23 | bwd_microstep: 217.50 | bwd_inner_microstep: 217.40 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 160.62 | bwd_microstep: 219.19 | bwd_inner_microstep: 219.11 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.91 | bwd_microstep: 223.31 | bwd_inner_microstep: 223.23 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.01 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.56 | bwd_microstep: 253.79 | bwd_inner_microstep: 243.89 | bwd_allreduce_microstep: 9.83 | step_microstep: 21.57 [Rank 0] time (ms) | fwd: 1294.88 | bwd: 1798.65 | bwd_inner: 1788.10 | bwd_allreduce: 10.21 | step: 22.08 8%|▊ | 239/2865 [1:34:01<3:08:21, 4.30s/it] {'loss': 0.9139, 'learning_rate': 3.970158369932879e-05, 'epoch': 0.25} 8%|▊ | 239/2865 [1:34:01<3:08:21, 4.30s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 162.85 | bwd_microstep: 242.31 | bwd_inner_microstep: 242.19 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 161.99 | bwd_microstep: 232.29 | bwd_inner_microstep: 232.06 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 162.15 | bwd_microstep: 250.11 | bwd_inner_microstep: 249.98 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 165.40 | bwd_microstep: 246.99 | bwd_inner_microstep: 246.59 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 164.89 | bwd_microstep: 254.42 | bwd_inner_microstep: 254.06 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 167.07 | bwd_microstep: 244.54 | bwd_inner_microstep: 244.41 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 164.66 | bwd_microstep: 257.34 | bwd_inner_microstep: 257.17 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.14 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 170.05 | bwd_microstep: 277.34 | bwd_inner_microstep: 266.75 | bwd_allreduce_microstep: 10.50 | step_microstep: 22.41 [Rank 0] time (ms) | fwd: 1318.90 | bwd: 2005.46 | bwd_inner: 1993.52 | bwd_allreduce: 11.19 | step: 23.10 8%|▊ | 240/2865 [1:34:04<2:58:01, 4.07s/it] {'loss': 0.904, 'learning_rate': 3.969767997416673e-05, 'epoch': 0.25} 8%|▊ | 240/2865 [1:34:04<2:58:01, 4.07s/it][INFO|trainer.py:3242] 2026-05-14 21:54:47,256 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 21:54:47,257 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 21:54:47,257 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-240 [INFO|configuration_utils.py:473] 2026-05-14 21:59:54,176 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-240/config.json [INFO|configuration_utils.py:594] 2026-05-14 21:59:54,185 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-240/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 21:59:59,630 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-240/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 21:59:59,641 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-240/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 21:59:59,649 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-240/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 21:59:59,651 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-240/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 22:00:08,121 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-165] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 303746.28 | bwd_microstep: 228.61 | bwd_inner_microstep: 228.45 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 163.87 | bwd_microstep: 231.22 | bwd_inner_microstep: 230.87 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 162.43 | bwd_microstep: 237.49 | bwd_inner_microstep: 237.10 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 162.54 | bwd_microstep: 229.61 | bwd_inner_microstep: 229.47 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 162.09 | bwd_microstep: 233.38 | bwd_inner_microstep: 232.98 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 166.65 | bwd_microstep: 228.14 | bwd_inner_microstep: 227.87 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.38 | bwd_microstep: 230.52 | bwd_inner_microstep: 230.14 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.05 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 161.84 | bwd_microstep: 294.70 | bwd_inner_microstep: 284.55 | bwd_allreduce_microstep: 10.07 | step_microstep: 21.82 [Rank 0] time (ms) | fwd: 304877.15 | bwd: 1913.85 | bwd_inner: 1901.84 | bwd_allreduce: 10.99 | step: 22.72 8%|▊ | 241/2865 [1:39:29<73:03:12, 100.23s/it] {'loss': 0.8896, 'learning_rate': 3.969375107584777e-05, 'epoch': 0.25} 8%|▊ | 241/2865 [1:39:29<73:03:12, 100.23s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 953 [Rank 0] time (ms) | fwd_microstep: 166.45 | bwd_microstep: 287.41 | bwd_inner_microstep: 287.05 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 163.91 | bwd_microstep: 233.05 | bwd_inner_microstep: 232.79 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 163.08 | bwd_microstep: 225.44 | bwd_inner_microstep: 225.11 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 163.07 | bwd_microstep: 224.16 | bwd_inner_microstep: 223.86 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 172.48 | bwd_microstep: 226.01 | bwd_inner_microstep: 225.65 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 147.54 | bwd_microstep: 197.78 | bwd_inner_microstep: 197.40 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.12 | bwd_microstep: 236.12 | bwd_inner_microstep: 235.98 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.94 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.31 | bwd_microstep: 262.83 | bwd_inner_microstep: 252.66 | bwd_allreduce_microstep: 10.07 | step_microstep: 21.46 [Rank 0] time (ms) | fwd: 1301.85 | bwd: 1892.96 | bwd_inner: 1880.97 | bwd_allreduce: 10.96 | step: 22.33 8%|▊ | 242/2865 [1:39:32<51:52:35, 71.20s/it] {'loss': 1.0579, 'learning_rate': 3.968979700939294e-05, 'epoch': 0.25} 8%|▊ | 242/2865 [1:39:32<51:52:35, 71.20s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 186.65 | bwd_microstep: 226.34 | bwd_inner_microstep: 226.06 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 173.89 | bwd_microstep: 225.64 | bwd_inner_microstep: 225.46 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.75 | bwd_microstep: 225.27 | bwd_inner_microstep: 224.97 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 162.41 | bwd_microstep: 227.56 | bwd_inner_microstep: 227.34 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 169.28 | bwd_microstep: 221.62 | bwd_inner_microstep: 221.26 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 150.74 | bwd_microstep: 194.59 | bwd_inner_microstep: 194.45 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 164.84 | bwd_microstep: 255.66 | bwd_inner_microstep: 255.40 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.32 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 187.40 | bwd_microstep: 269.04 | bwd_inner_microstep: 253.55 | bwd_allreduce_microstep: 15.38 | step_microstep: 26.41 [Rank 0] time (ms) | fwd: 1356.79 | bwd: 1845.87 | bwd_inner: 1828.90 | bwd_allreduce: 16.07 | step: 27.24 8%|▊ | 243/2865 [1:39:36<37:02:51, 50.87s/it] {'loss': 1.0181, 'learning_rate': 3.9685817779855434e-05, 'epoch': 0.25} 8%|▊ | 243/2865 [1:39:36<37:02:51, 50.87s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 179.53 | bwd_microstep: 224.46 | bwd_inner_microstep: 224.33 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 168.67 | bwd_microstep: 226.92 | bwd_inner_microstep: 226.69 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 162.44 | bwd_microstep: 226.94 | bwd_inner_microstep: 226.56 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.50 | bwd_microstep: 229.04 | bwd_inner_microstep: 228.56 | bwd_allreduce_microstep: 0.22 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 165.60 | bwd_microstep: 232.27 | bwd_inner_microstep: 231.86 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 161.67 | bwd_microstep: 231.50 | bwd_inner_microstep: 231.09 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 165.38 | bwd_microstep: 230.41 | bwd_inner_microstep: 230.01 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.95 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 149.50 | bwd_microstep: 236.15 | bwd_inner_microstep: 226.19 | bwd_allreduce_microstep: 9.88 | step_microstep: 21.55 [Rank 0] time (ms) | fwd: 1314.14 | bwd: 1837.91 | bwd_inner: 1825.73 | bwd_allreduce: 10.99 | step: 22.58 9%|▊ | 244/2865 [1:39:39<26:39:21, 36.61s/it] {'loss': 0.9848, 'learning_rate': 3.968181339232062e-05, 'epoch': 0.26} 9%|▊ | 244/2865 [1:39:39<26:39:21, 36.61s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 163.75 | bwd_microstep: 226.75 | bwd_inner_microstep: 226.54 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 161.99 | bwd_microstep: 226.84 | bwd_inner_microstep: 226.60 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 174.92 | bwd_microstep: 228.05 | bwd_inner_microstep: 227.95 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 167.18 | bwd_microstep: 228.44 | bwd_inner_microstep: 228.31 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.62 | bwd_microstep: 227.84 | bwd_inner_microstep: 227.70 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 165.56 | bwd_microstep: 229.89 | bwd_inner_microstep: 229.54 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 165.07 | bwd_microstep: 224.36 | bwd_inner_microstep: 224.24 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 147.36 | bwd_microstep: 231.97 | bwd_inner_microstep: 221.53 | bwd_allreduce_microstep: 10.28 | step_microstep: 21.28 [Rank 0] time (ms) | fwd: 1307.31 | bwd: 1824.29 | bwd_inner: 1812.69 | bwd_allreduce: 10.84 | step: 21.97 9%|▊ | 245/2865 [1:39:42<19:22:27, 26.62s/it] {'loss': 1.1867, 'learning_rate': 3.967778385190601e-05, 'epoch': 0.26} 9%|▊ | 245/2865 [1:39:42<19:22:27, 26.62s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 161.84 | bwd_microstep: 228.53 | bwd_inner_microstep: 228.07 | bwd_allreduce_microstep: 0.20 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 163.91 | bwd_microstep: 232.59 | bwd_inner_microstep: 232.44 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 162.31 | bwd_microstep: 267.17 | bwd_inner_microstep: 266.99 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 163.28 | bwd_microstep: 252.16 | bwd_inner_microstep: 251.85 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.19 | bwd_microstep: 225.86 | bwd_inner_microstep: 225.47 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 163.09 | bwd_microstep: 223.57 | bwd_inner_microstep: 223.31 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 160.84 | bwd_microstep: 221.36 | bwd_inner_microstep: 221.17 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.39 | bwd_microstep: 252.57 | bwd_inner_microstep: 242.65 | bwd_allreduce_microstep: 9.84 | step_microstep: 21.07 [Rank 0] time (ms) | fwd: 1299.75 | bwd: 1903.98 | bwd_inner: 1892.33 | bwd_allreduce: 10.66 | step: 21.97 9%|▊ | 246/2865 [1:39:46<14:17:38, 19.65s/it] {'loss': 0.9887, 'learning_rate': 3.967372916376125e-05, 'epoch': 0.26} 9%|▊ | 246/2865 [1:39:46<14:17:38, 19.65s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 164.44 | bwd_microstep: 214.44 | bwd_inner_microstep: 214.24 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 189.98 | bwd_microstep: 217.85 | bwd_inner_microstep: 217.74 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 162.38 | bwd_microstep: 237.04 | bwd_inner_microstep: 236.89 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 160.29 | bwd_microstep: 238.78 | bwd_inner_microstep: 238.65 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.17 | bwd_microstep: 217.85 | bwd_inner_microstep: 217.68 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 163.59 | bwd_microstep: 219.87 | bwd_inner_microstep: 219.71 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.54 | bwd_microstep: 215.97 | bwd_inner_microstep: 215.86 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.20 | optimizer_step: 4.86 [Rank 0] time (ms) | fwd_microstep: 151.53 | bwd_microstep: 251.61 | bwd_inner_microstep: 237.57 | bwd_allreduce_microstep: 13.94 | step_microstep: 24.14 [Rank 0] time (ms) | fwd: 1313.82 | bwd: 1813.52 | bwd_inner: 1798.49 | bwd_allreduce: 14.43 | step: 24.71 9%|▊ | 247/2865 [1:39:49<10:43:09, 14.74s/it] {'loss': 0.9262, 'learning_rate': 3.9669649333068145e-05, 'epoch': 0.26} 9%|▊ | 247/2865 [1:39:49<10:43:09, 14.74s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 162.38 | bwd_microstep: 220.42 | bwd_inner_microstep: 220.06 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 161.96 | bwd_microstep: 224.70 | bwd_inner_microstep: 224.56 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 163.10 | bwd_microstep: 226.22 | bwd_inner_microstep: 226.11 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 160.85 | bwd_microstep: 234.03 | bwd_inner_microstep: 233.89 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.31 | bwd_microstep: 215.06 | bwd_inner_microstep: 214.76 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 160.92 | bwd_microstep: 217.38 | bwd_inner_microstep: 217.08 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 161.15 | bwd_microstep: 221.21 | bwd_inner_microstep: 220.88 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 162.08 | bwd_microstep: 249.99 | bwd_inner_microstep: 239.96 | bwd_allreduce_microstep: 9.86 | step_microstep: 21.08 [Rank 0] time (ms) | fwd: 1293.63 | bwd: 1809.12 | bwd_inner: 1797.69 | bwd_allreduce: 10.55 | step: 21.80 9%|▊ | 248/2865 [1:39:52<8:12:48, 11.30s/it] {'loss': 1.1448, 'learning_rate': 3.96655443650406e-05, 'epoch': 0.26} 9%|▊ | 248/2865 [1:39:52<8:12:48, 11.30s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 946 [Rank 0] time (ms) | fwd_microstep: 162.10 | bwd_microstep: 217.32 | bwd_inner_microstep: 216.98 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 161.73 | bwd_microstep: 222.06 | bwd_inner_microstep: 221.68 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 162.50 | bwd_microstep: 221.60 | bwd_inner_microstep: 221.20 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.09 | bwd_microstep: 225.48 | bwd_inner_microstep: 225.19 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.20 | bwd_microstep: 215.73 | bwd_inner_microstep: 215.36 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.63 | bwd_microstep: 216.93 | bwd_inner_microstep: 216.80 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.11 | bwd_microstep: 214.49 | bwd_inner_microstep: 214.30 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 160.50 | bwd_microstep: 252.60 | bwd_inner_microstep: 242.74 | bwd_allreduce_microstep: 9.78 | step_microstep: 21.35 [Rank 0] time (ms) | fwd: 1290.71 | bwd: 1786.37 | bwd_inner: 1774.74 | bwd_allreduce: 10.68 | step: 22.29 9%|▊ | 249/2865 [1:39:55<6:27:35, 8.89s/it] {'loss': 0.94, 'learning_rate': 3.966141426492468e-05, 'epoch': 0.26} 9%|▊ | 249/2865 [1:39:55<6:27:35, 8.89s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 161.55 | bwd_microstep: 216.48 | bwd_inner_microstep: 216.23 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 161.45 | bwd_microstep: 218.67 | bwd_inner_microstep: 218.28 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 162.45 | bwd_microstep: 220.34 | bwd_inner_microstep: 220.20 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 163.00 | bwd_microstep: 217.53 | bwd_inner_microstep: 217.16 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.54 | bwd_microstep: 247.54 | bwd_inner_microstep: 247.18 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.89 | bwd_microstep: 193.67 | bwd_inner_microstep: 193.39 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.80 | bwd_microstep: 191.28 | bwd_inner_microstep: 190.89 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.69 | bwd_microstep: 219.12 | bwd_inner_microstep: 208.51 | bwd_allreduce_microstep: 10.51 | step_microstep: 21.16 [Rank 0] time (ms) | fwd: 1294.22 | bwd: 1724.83 | bwd_inner: 1712.34 | bwd_allreduce: 11.38 | step: 22.20 9%|▊ | 250/2865 [1:39:59<5:12:50, 7.18s/it] {'loss': 1.0778, 'learning_rate': 3.9657259037998554e-05, 'epoch': 0.26} 9%|▊ | 250/2865 [1:39:59<5:12:50, 7.18s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 161.78 | bwd_microstep: 212.31 | bwd_inner_microstep: 211.91 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 163.75 | bwd_microstep: 211.91 | bwd_inner_microstep: 211.70 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 161.77 | bwd_microstep: 215.46 | bwd_inner_microstep: 215.13 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 161.15 | bwd_microstep: 213.22 | bwd_inner_microstep: 212.92 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 160.62 | bwd_microstep: 215.23 | bwd_inner_microstep: 215.09 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.15 | bwd_microstep: 215.39 | bwd_inner_microstep: 215.25 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.96 | bwd_microstep: 217.03 | bwd_inner_microstep: 216.72 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 164.40 | bwd_microstep: 248.92 | bwd_inner_microstep: 238.65 | bwd_allreduce_microstep: 10.10 | step_microstep: 21.18 [Rank 0] time (ms) | fwd: 1296.48 | bwd: 1749.58 | bwd_inner: 1737.81 | bwd_allreduce: 10.77 | step: 21.82 9%|▉ | 251/2865 [1:40:02<4:20:51, 5.99s/it] {'loss': 1.0115, 'learning_rate': 3.965307868957249e-05, 'epoch': 0.26} 9%|▉ | 251/2865 [1:40:02<4:20:51, 5.99s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 162.60 | bwd_microstep: 218.62 | bwd_inner_microstep: 218.18 | bwd_allreduce_microstep: 0.20 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 160.93 | bwd_microstep: 220.74 | bwd_inner_microstep: 220.35 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 160.98 | bwd_microstep: 221.11 | bwd_inner_microstep: 221.00 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.05 | bwd_microstep: 221.89 | bwd_inner_microstep: 221.62 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.19 | bwd_microstep: 213.28 | bwd_inner_microstep: 212.97 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.35 | bwd_microstep: 216.33 | bwd_inner_microstep: 216.19 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 166.33 | bwd_microstep: 213.01 | bwd_inner_microstep: 212.69 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.22 | bwd_microstep: 251.33 | bwd_inner_microstep: 239.72 | bwd_allreduce_microstep: 11.44 | step_microstep: 21.09 [Rank 0] time (ms) | fwd: 1295.55 | bwd: 1776.44 | bwd_inner: 1763.21 | bwd_allreduce: 12.26 | step: 21.74 9%|▉ | 252/2865 [1:40:05<3:44:43, 5.16s/it] {'loss': 0.8901, 'learning_rate': 3.964887322498888e-05, 'epoch': 0.26} 9%|▉ | 252/2865 [1:40:05<3:44:43, 5.16s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.04 | bwd_microstep: 214.49 | bwd_inner_microstep: 214.11 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.88 | bwd_microstep: 216.54 | bwd_inner_microstep: 216.42 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 160.94 | bwd_microstep: 216.52 | bwd_inner_microstep: 216.15 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 160.66 | bwd_microstep: 214.76 | bwd_inner_microstep: 214.62 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 147.11 | bwd_microstep: 174.37 | bwd_inner_microstep: 174.04 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.14 | bwd_microstep: 191.50 | bwd_inner_microstep: 191.38 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.86 | bwd_microstep: 217.17 | bwd_inner_microstep: 217.04 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 909 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 161.02 | bwd_microstep: 244.46 | bwd_inner_microstep: 234.66 | bwd_allreduce_microstep: 9.65 | step_microstep: 21.34 [Rank 0] time (ms) | fwd: 1274.56 | bwd: 1689.95 | bwd_inner: 1678.78 | bwd_allreduce: 10.29 | step: 22.02 9%|▉ | 253/2865 [1:40:08<3:18:01, 4.55s/it] {'loss': 1.1599, 'learning_rate': 3.9644642649622207e-05, 'epoch': 0.26} 9%|▉ | 253/2865 [1:40:08<3:18:01, 4.55s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 161.10 | bwd_microstep: 212.83 | bwd_inner_microstep: 212.47 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.76 | bwd_microstep: 215.17 | bwd_inner_microstep: 214.87 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 162.17 | bwd_microstep: 218.41 | bwd_inner_microstep: 218.04 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.97 | bwd_microstep: 213.34 | bwd_inner_microstep: 213.22 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.74 | bwd_microstep: 217.46 | bwd_inner_microstep: 217.16 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.54 | bwd_microstep: 219.66 | bwd_inner_microstep: 219.28 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 161.51 | bwd_microstep: 218.55 | bwd_inner_microstep: 218.16 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.58 | bwd_microstep: 263.97 | bwd_inner_microstep: 250.04 | bwd_allreduce_microstep: 13.78 | step_microstep: 23.43 [Rank 0] time (ms) | fwd: 1292.26 | bwd: 1779.57 | bwd_inner: 1763.70 | bwd_allreduce: 14.74 | step: 24.41 9%|▉ | 254/2865 [1:40:11<3:01:00, 4.16s/it] {'loss': 1.0352, 'learning_rate': 3.9640386968879046e-05, 'epoch': 0.27} 9%|▉ | 254/2865 [1:40:11<3:01:00, 4.16s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 944 [Rank 0] time (ms) | fwd_microstep: 163.00 | bwd_microstep: 219.89 | bwd_inner_microstep: 219.59 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 161.91 | bwd_microstep: 219.37 | bwd_inner_microstep: 219.20 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 160.86 | bwd_microstep: 215.87 | bwd_inner_microstep: 215.56 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 162.15 | bwd_microstep: 215.41 | bwd_inner_microstep: 215.12 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.82 | bwd_microstep: 215.60 | bwd_inner_microstep: 215.22 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 160.67 | bwd_microstep: 219.08 | bwd_inner_microstep: 218.73 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.53 | bwd_microstep: 216.99 | bwd_inner_microstep: 216.86 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 162.39 | bwd_microstep: 275.97 | bwd_inner_microstep: 266.07 | bwd_allreduce_microstep: 9.71 | step_microstep: 21.34 [Rank 0] time (ms) | fwd: 1293.21 | bwd: 1798.32 | bwd_inner: 1786.85 | bwd_allreduce: 10.48 | step: 22.17 9%|▉ | 255/2865 [1:40:15<2:49:15, 3.89s/it] {'loss': 0.8569, 'learning_rate': 3.963610618819804e-05, 'epoch': 0.27} 9%|▉ | 255/2865 [1:40:15<2:49:15, 3.89s/it][INFO|trainer.py:3242] 2026-05-14 22:00:57,866 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 22:00:57,866 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 22:00:57,866 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-255 [INFO|configuration_utils.py:473] 2026-05-14 22:06:04,671 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-255/config.json [INFO|configuration_utils.py:594] 2026-05-14 22:06:04,679 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-255/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 22:06:09,490 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-255/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 22:06:09,508 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-255/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 22:06:09,521 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-255/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 22:06:09,523 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-255/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 22:06:19,948 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-180] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 303269.79 | bwd_microstep: 252.09 | bwd_inner_microstep: 251.83 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 199.81 | bwd_microstep: 283.98 | bwd_inner_microstep: 283.64 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 208.89 | bwd_microstep: 230.92 | bwd_inner_microstep: 230.58 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 189.29 | bwd_microstep: 273.99 | bwd_inner_microstep: 273.83 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 200.96 | bwd_microstep: 336.85 | bwd_inner_microstep: 336.53 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 190.79 | bwd_microstep: 267.28 | bwd_inner_microstep: 267.12 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 170.36 | bwd_microstep: 280.27 | bwd_inner_microstep: 280.11 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.94 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 190.07 | bwd_microstep: 299.41 | bwd_inner_microstep: 288.04 | bwd_allreduce_microstep: 11.24 | step_microstep: 21.69 [Rank 0] time (ms) | fwd: 304608.60 | bwd: 2224.93 | bwd_inner: 2212.03 | bwd_allreduce: 11.98 | step: 22.57 9%|▉ | 256/2865 [1:45:41<73:01:19, 100.76s/it] {'loss': 0.9353, 'learning_rate': 3.963180031304993e-05, 'epoch': 0.27} 9%|▉ | 256/2865 [1:45:41<73:01:19, 100.76s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 175.69 | bwd_microstep: 268.58 | bwd_inner_microstep: 268.16 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 180.01 | bwd_microstep: 273.73 | bwd_inner_microstep: 273.57 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 169.78 | bwd_microstep: 247.99 | bwd_inner_microstep: 247.70 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 173.60 | bwd_microstep: 274.45 | bwd_inner_microstep: 274.30 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 190.95 | bwd_microstep: 289.23 | bwd_inner_microstep: 289.03 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 166.93 | bwd_microstep: 224.11 | bwd_inner_microstep: 224.05 | bwd_allreduce_microstep: 0.03 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 162.35 | bwd_microstep: 222.51 | bwd_inner_microstep: 222.35 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 163.93 | bwd_microstep: 289.55 | bwd_inner_microstep: 277.44 | bwd_allreduce_microstep: 11.91 | step_microstep: 21.21 [Rank 0] time (ms) | fwd: 1383.14 | bwd: 2090.28 | bwd_inner: 2076.95 | bwd_allreduce: 12.51 | step: 22.08 9%|▉ | 257/2865 [1:45:45<51:53:22, 71.63s/it] {'loss': 1.1608, 'learning_rate': 3.962746934893753e-05, 'epoch': 0.27} 9%|▉ | 257/2865 [1:45:45<51:53:22, 71.63s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 173.15 | bwd_microstep: 226.41 | bwd_inner_microstep: 226.30 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 172.10 | bwd_microstep: 238.21 | bwd_inner_microstep: 237.91 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 162.02 | bwd_microstep: 231.06 | bwd_inner_microstep: 230.85 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 167.98 | bwd_microstep: 226.57 | bwd_inner_microstep: 226.49 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 163.36 | bwd_microstep: 230.01 | bwd_inner_microstep: 229.73 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 164.32 | bwd_microstep: 233.56 | bwd_inner_microstep: 233.37 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 171.54 | bwd_microstep: 228.38 | bwd_inner_microstep: 227.99 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 168.37 | bwd_microstep: 265.68 | bwd_inner_microstep: 255.72 | bwd_allreduce_microstep: 9.85 | step_microstep: 21.86 [Rank 0] time (ms) | fwd: 1342.74 | bwd: 1880.01 | bwd_inner: 1868.70 | bwd_allreduce: 10.50 | step: 22.60 9%|▉ | 258/2865 [1:45:49<37:03:10, 51.17s/it] {'loss': 0.9367, 'learning_rate': 3.962311330139569e-05, 'epoch': 0.27} 9%|▉ | 258/2865 [1:45:49<37:03:10, 51.17s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 174.21 | bwd_microstep: 245.11 | bwd_inner_microstep: 242.99 | bwd_allreduce_microstep: 1.96 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 167.77 | bwd_microstep: 247.19 | bwd_inner_microstep: 247.06 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 166.12 | bwd_microstep: 232.32 | bwd_inner_microstep: 232.16 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 162.75 | bwd_microstep: 232.38 | bwd_inner_microstep: 232.20 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 162.90 | bwd_microstep: 229.63 | bwd_inner_microstep: 229.50 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 167.02 | bwd_microstep: 232.79 | bwd_inner_microstep: 232.65 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 188.13 | bwd_microstep: 244.60 | bwd_inner_microstep: 244.40 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 168.68 | bwd_microstep: 275.07 | bwd_inner_microstep: 262.44 | bwd_allreduce_microstep: 12.53 | step_microstep: 21.29 [Rank 0] time (ms) | fwd: 1357.47 | bwd: 1939.21 | bwd_inner: 1923.64 | bwd_allreduce: 14.95 | step: 21.91 9%|▉ | 259/2865 [1:45:52<26:41:04, 36.86s/it] {'loss': 1.0189, 'learning_rate': 3.9618732175991344e-05, 'epoch': 0.27} 9%|▉ | 259/2865 [1:45:52<26:41:04, 36.86s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 164.90 | bwd_microstep: 260.82 | bwd_inner_microstep: 260.71 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 167.41 | bwd_microstep: 239.35 | bwd_inner_microstep: 239.23 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 163.12 | bwd_microstep: 248.58 | bwd_inner_microstep: 248.45 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 164.20 | bwd_microstep: 234.46 | bwd_inner_microstep: 234.19 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 165.59 | bwd_microstep: 242.04 | bwd_inner_microstep: 241.77 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 162.84 | bwd_microstep: 239.13 | bwd_inner_microstep: 238.97 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 155.10 | bwd_microstep: 209.64 | bwd_inner_microstep: 209.29 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 158.35 | bwd_microstep: 256.35 | bwd_inner_microstep: 246.37 | bwd_allreduce_microstep: 9.88 | step_microstep: 21.60 [Rank 0] time (ms) | fwd: 1301.40 | bwd: 1930.51 | bwd_inner: 1919.26 | bwd_allreduce: 10.50 | step: 22.32 9%|▉ | 260/2865 [1:45:55<19:24:41, 26.83s/it] {'loss': 0.8885, 'learning_rate': 3.9614325978323453e-05, 'epoch': 0.27} 9%|▉ | 260/2865 [1:45:55<19:24:41, 26.83s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 171.64 | bwd_microstep: 224.18 | bwd_inner_microstep: 224.04 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 938 [Rank 0] time (ms) | fwd_microstep: 169.69 | bwd_microstep: 231.07 | bwd_inner_microstep: 230.84 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 182.83 | bwd_microstep: 281.72 | bwd_inner_microstep: 281.52 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 186.39 | bwd_microstep: 271.84 | bwd_inner_microstep: 271.67 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 200.20 | bwd_microstep: 250.11 | bwd_inner_microstep: 249.80 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 181.55 | bwd_microstep: 226.75 | bwd_inner_microstep: 226.58 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 171.37 | bwd_microstep: 226.52 | bwd_inner_microstep: 226.15 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.02 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 165.52 | bwd_microstep: 317.05 | bwd_inner_microstep: 305.79 | bwd_allreduce_microstep: 11.16 | step_microstep: 22.02 [Rank 0] time (ms) | fwd: 1429.03 | bwd: 2029.35 | bwd_inner: 2016.75 | bwd_allreduce: 11.87 | step: 22.79 9%|▉ | 261/2865 [1:45:59<14:22:55, 19.88s/it] {'loss': 0.8577, 'learning_rate': 3.960989471402305e-05, 'epoch': 0.27} 9%|▉ | 261/2865 [1:45:59<14:22:55, 19.88s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 182.91 | bwd_microstep: 224.21 | bwd_inner_microstep: 224.00 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 169.88 | bwd_microstep: 228.66 | bwd_inner_microstep: 228.45 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 186.98 | bwd_microstep: 263.12 | bwd_inner_microstep: 262.85 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 198.60 | bwd_microstep: 264.40 | bwd_inner_microstep: 264.01 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.28 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 191.51 | bwd_microstep: 235.92 | bwd_inner_microstep: 235.44 | bwd_allreduce_microstep: 0.21 | step_microstep: 0.26 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 192.39 | bwd_microstep: 281.84 | bwd_inner_microstep: 281.41 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.25 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 164.33 | bwd_microstep: 227.73 | bwd_inner_microstep: 227.43 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.09 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 199.89 | bwd_microstep: 311.99 | bwd_inner_microstep: 299.33 | bwd_allreduce_microstep: 12.47 | step_microstep: 23.41 [Rank 0] time (ms) | fwd: 1486.34 | bwd: 2038.09 | bwd_inner: 2023.43 | bwd_allreduce: 13.40 | step: 24.60 9%|▉ | 262/2865 [1:46:03<10:52:33, 15.04s/it] {'loss': 1.1157, 'learning_rate': 3.960543838875317e-05, 'epoch': 0.27} 9%|▉ | 262/2865 [1:46:03<10:52:33, 15.04s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 181.30 | bwd_microstep: 297.21 | bwd_inner_microstep: 296.97 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 167.58 | bwd_microstep: 253.91 | bwd_inner_microstep: 253.62 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 173.58 | bwd_microstep: 261.94 | bwd_inner_microstep: 261.78 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 184.66 | bwd_microstep: 300.33 | bwd_inner_microstep: 299.98 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 189.68 | bwd_microstep: 267.15 | bwd_inner_microstep: 266.98 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 908 [Rank 0] time (ms) | fwd_microstep: 199.98 | bwd_microstep: 267.19 | bwd_inner_microstep: 266.60 | bwd_allreduce_microstep: 0.23 | step_microstep: 0.31 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 170.34 | bwd_microstep: 257.20 | bwd_inner_microstep: 256.72 | bwd_allreduce_microstep: 0.23 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.16 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 164.19 | bwd_microstep: 318.39 | bwd_inner_microstep: 306.05 | bwd_allreduce_microstep: 12.24 | step_microstep: 23.32 [Rank 0] time (ms) | fwd: 1431.14 | bwd: 2223.49 | bwd_inner: 2209.16 | bwd_allreduce: 13.17 | step: 24.25 9%|▉ | 263/2865 [1:46:07<8:26:52, 11.69s/it] {'loss': 0.7711, 'learning_rate': 3.960095700820888e-05, 'epoch': 0.28} 9%|▉ | 263/2865 [1:46:07<8:26:52, 11.69s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 191.64 | bwd_microstep: 277.02 | bwd_inner_microstep: 276.82 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 186.79 | bwd_microstep: 275.58 | bwd_inner_microstep: 275.32 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 186.90 | bwd_microstep: 277.59 | bwd_inner_microstep: 277.40 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 186.15 | bwd_microstep: 276.34 | bwd_inner_microstep: 276.12 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 188.48 | bwd_microstep: 240.16 | bwd_inner_microstep: 239.81 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.24 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 191.57 | bwd_microstep: 272.99 | bwd_inner_microstep: 272.76 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 202.66 | bwd_microstep: 276.87 | bwd_inner_microstep: 276.72 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.99 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 180.00 | bwd_microstep: 323.92 | bwd_inner_microstep: 309.78 | bwd_allreduce_microstep: 14.04 | step_microstep: 22.08 [Rank 0] time (ms) | fwd: 1514.02 | bwd: 2220.68 | bwd_inner: 2205.04 | bwd_allreduce: 14.77 | step: 23.04 9%|▉ | 264/2865 [1:46:11<6:46:02, 9.37s/it] {'loss': 0.9706, 'learning_rate': 3.9596450578117296e-05, 'epoch': 0.28} 9%|▉ | 264/2865 [1:46:11<6:46:02, 9.37s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 170.96 | bwd_microstep: 245.55 | bwd_inner_microstep: 245.39 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 193.45 | bwd_microstep: 297.92 | bwd_inner_microstep: 297.75 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 182.33 | bwd_microstep: 308.61 | bwd_inner_microstep: 308.43 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 175.09 | bwd_microstep: 334.00 | bwd_inner_microstep: 333.81 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 157.89 | bwd_microstep: 247.81 | bwd_inner_microstep: 247.50 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 166.71 | bwd_microstep: 284.18 | bwd_inner_microstep: 283.98 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 169.62 | bwd_microstep: 246.87 | bwd_inner_microstep: 246.73 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.01 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 168.55 | bwd_microstep: 297.04 | bwd_inner_microstep: 283.28 | bwd_allreduce_microstep: 13.63 | step_microstep: 21.73 [Rank 0] time (ms) | fwd: 1384.47 | bwd: 2262.11 | bwd_inner: 2247.19 | bwd_allreduce: 14.25 | step: 22.52 9%|▉ | 265/2865 [1:46:15<5:34:09, 7.71s/it] {'loss': 0.9, 'learning_rate': 3.9591919104237516e-05, 'epoch': 0.28} 9%|▉ | 265/2865 [1:46:15<5:34:09, 7.71s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 180.51 | bwd_microstep: 295.44 | bwd_inner_microstep: 295.23 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 180.59 | bwd_microstep: 310.25 | bwd_inner_microstep: 309.98 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 213.07 | bwd_microstep: 251.10 | bwd_inner_microstep: 250.83 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 174.81 | bwd_microstep: 255.32 | bwd_inner_microstep: 255.18 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 164.71 | bwd_microstep: 270.37 | bwd_inner_microstep: 270.15 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 183.68 | bwd_microstep: 235.62 | bwd_inner_microstep: 235.31 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 164.23 | bwd_microstep: 253.81 | bwd_inner_microstep: 253.38 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.08 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 183.63 | bwd_microstep: 322.35 | bwd_inner_microstep: 310.81 | bwd_allreduce_microstep: 11.34 | step_microstep: 21.93 [Rank 0] time (ms) | fwd: 1445.10 | bwd: 2194.39 | bwd_inner: 2181.29 | bwd_allreduce: 12.13 | step: 22.72 9%|▉ | 266/2865 [1:46:18<4:43:55, 6.55s/it] {'loss': 1.0676, 'learning_rate': 3.958736259236065e-05, 'epoch': 0.28} 9%|▉ | 266/2865 [1:46:18<4:43:55, 6.55s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 176.18 | bwd_microstep: 277.68 | bwd_inner_microstep: 277.30 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 177.77 | bwd_microstep: 266.18 | bwd_inner_microstep: 265.98 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 163.74 | bwd_microstep: 249.29 | bwd_inner_microstep: 248.95 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 183.40 | bwd_microstep: 250.10 | bwd_inner_microstep: 249.96 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 170.91 | bwd_microstep: 257.55 | bwd_inner_microstep: 256.98 | bwd_allreduce_microstep: 0.22 | step_microstep: 0.31 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 168.78 | bwd_microstep: 238.22 | bwd_inner_microstep: 237.83 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 164.45 | bwd_microstep: 230.70 | bwd_inner_microstep: 230.40 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.94 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 175.78 | bwd_microstep: 264.42 | bwd_inner_microstep: 254.11 | bwd_allreduce_microstep: 10.23 | step_microstep: 21.28 [Rank 0] time (ms) | fwd: 1380.86 | bwd: 2034.31 | bwd_inner: 2022.04 | bwd_allreduce: 11.17 | step: 22.27 9%|▉ | 267/2865 [1:46:22<4:05:52, 5.68s/it] {'loss': 0.8822, 'learning_rate': 3.958278104830982e-05, 'epoch': 0.28} 9%|▉ | 267/2865 [1:46:22<4:05:52, 5.68s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 171.31 | bwd_microstep: 240.66 | bwd_inner_microstep: 240.48 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 179.50 | bwd_microstep: 275.97 | bwd_inner_microstep: 275.84 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.83 | bwd_microstep: 226.82 | bwd_inner_microstep: 226.57 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 162.98 | bwd_microstep: 229.88 | bwd_inner_microstep: 229.53 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 161.36 | bwd_microstep: 223.26 | bwd_inner_microstep: 222.91 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 161.99 | bwd_microstep: 236.44 | bwd_inner_microstep: 236.10 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 162.58 | bwd_microstep: 238.25 | bwd_inner_microstep: 238.10 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.25 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.13 | bwd_microstep: 266.70 | bwd_inner_microstep: 256.72 | bwd_allreduce_microstep: 9.76 | step_microstep: 21.87 [Rank 0] time (ms) | fwd: 1323.56 | bwd: 1938.09 | bwd_inner: 1926.66 | bwd_allreduce: 10.56 | step: 22.50 9%|▉ | 268/2865 [1:46:25<3:36:49, 5.01s/it] {'loss': 0.9276, 'learning_rate': 3.957817447794011e-05, 'epoch': 0.28} 9%|▉ | 268/2865 [1:46:25<3:36:49, 5.01s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 164.32 | bwd_microstep: 253.04 | bwd_inner_microstep: 252.90 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 166.89 | bwd_microstep: 231.64 | bwd_inner_microstep: 231.43 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 166.40 | bwd_microstep: 220.75 | bwd_inner_microstep: 220.37 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 165.45 | bwd_microstep: 220.45 | bwd_inner_microstep: 220.16 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 172.10 | bwd_microstep: 227.66 | bwd_inner_microstep: 227.33 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 163.83 | bwd_microstep: 226.44 | bwd_inner_microstep: 225.86 | bwd_allreduce_microstep: 0.22 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 161.98 | bwd_microstep: 229.67 | bwd_inner_microstep: 229.32 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.90 | bwd_microstep: 256.06 | bwd_inner_microstep: 245.03 | bwd_allreduce_microstep: 10.86 | step_microstep: 21.41 [Rank 0] time (ms) | fwd: 1321.74 | bwd: 1865.92 | bwd_inner: 1852.95 | bwd_allreduce: 11.72 | step: 22.16 9%|▉ | 269/2865 [1:46:29<3:15:26, 4.52s/it] {'loss': 0.9745, 'learning_rate': 3.957354288713863e-05, 'epoch': 0.28} 9%|▉ | 269/2865 [1:46:29<3:15:26, 4.52s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 163.34 | bwd_microstep: 227.73 | bwd_inner_microstep: 227.48 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 168.02 | bwd_microstep: 232.59 | bwd_inner_microstep: 232.35 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 162.55 | bwd_microstep: 269.37 | bwd_inner_microstep: 268.97 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 164.75 | bwd_microstep: 234.62 | bwd_inner_microstep: 234.42 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 168.98 | bwd_microstep: 257.28 | bwd_inner_microstep: 256.94 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 164.09 | bwd_microstep: 250.70 | bwd_inner_microstep: 250.52 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 185.12 | bwd_microstep: 241.08 | bwd_inner_microstep: 240.89 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.32 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 172.50 | bwd_microstep: 286.14 | bwd_inner_microstep: 271.96 | bwd_allreduce_microstep: 14.06 | step_microstep: 25.06 [Rank 0] time (ms) | fwd: 1349.19 | bwd: 1999.64 | bwd_inner: 1983.95 | bwd_allreduce: 14.78 | step: 25.79 9%|▉ | 270/2865 [1:46:32<3:02:53, 4.23s/it] {'loss': 0.8937, 'learning_rate': 3.956888628182442e-05, 'epoch': 0.28} 9%|▉ | 270/2865 [1:46:32<3:02:53, 4.23s/it][INFO|trainer.py:3242] 2026-05-14 22:07:15,572 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 22:07:15,573 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 22:07:15,573 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-270 [INFO|configuration_utils.py:473] 2026-05-14 22:12:22,789 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-270/config.json [INFO|configuration_utils.py:594] 2026-05-14 22:12:22,808 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-270/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 22:12:29,065 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-270/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 22:12:29,079 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-270/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 22:12:29,089 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-270/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 22:12:29,092 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-270/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 22:12:39,564 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-195] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 303293.69 | bwd_microstep: 257.61 | bwd_inner_microstep: 257.45 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 165.85 | bwd_microstep: 258.32 | bwd_inner_microstep: 258.07 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 199.16 | bwd_microstep: 247.30 | bwd_inner_microstep: 247.06 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 204.88 | bwd_microstep: 304.40 | bwd_inner_microstep: 303.98 | bwd_allreduce_microstep: 0.20 | step_microstep: 0.24 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 172.56 | bwd_microstep: 265.45 | bwd_inner_microstep: 265.15 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 200.04 | bwd_microstep: 278.40 | bwd_inner_microstep: 278.12 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 195.46 | bwd_microstep: 280.50 | bwd_inner_microstep: 280.33 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.96 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 186.75 | bwd_microstep: 272.87 | bwd_inner_microstep: 260.86 | bwd_allreduce_microstep: 11.90 | step_microstep: 21.43 [Rank 0] time (ms) | fwd: 304607.60 | bwd: 2165.00 | bwd_inner: 2151.38 | bwd_allreduce: 12.72 | step: 22.24 9%|▉ | 271/2865 [1:52:01<73:07:26, 101.48s/it] {'loss': 0.8733, 'learning_rate': 3.9564204667948506e-05, 'epoch': 0.28} 9%|▉ | 271/2865 [1:52:01<73:07:26, 101.48s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 951 [Rank 0] time (ms) | fwd_microstep: 167.56 | bwd_microstep: 234.27 | bwd_inner_microstep: 234.15 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 167.01 | bwd_microstep: 245.81 | bwd_inner_microstep: 245.66 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 175.00 | bwd_microstep: 275.87 | bwd_inner_microstep: 275.50 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 182.71 | bwd_microstep: 253.42 | bwd_inner_microstep: 253.29 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 165.55 | bwd_microstep: 248.91 | bwd_inner_microstep: 248.77 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 168.48 | bwd_microstep: 245.84 | bwd_inner_microstep: 245.53 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 170.28 | bwd_microstep: 235.89 | bwd_inner_microstep: 235.58 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.96 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 170.85 | bwd_microstep: 309.30 | bwd_inner_microstep: 298.28 | bwd_allreduce_microstep: 10.81 | step_microstep: 23.00 [Rank 0] time (ms) | fwd: 1367.31 | bwd: 2049.43 | bwd_inner: 2037.12 | bwd_allreduce: 11.48 | step: 23.72 9%|▉ | 272/2865 [1:52:04<51:56:59, 72.12s/it] {'loss': 0.9085, 'learning_rate': 3.95594980514939e-05, 'epoch': 0.28} 9%|▉ | 272/2865 [1:52:04<51:56:59, 72.12s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 163.78 | bwd_microstep: 257.85 | bwd_inner_microstep: 257.58 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 179.36 | bwd_microstep: 298.71 | bwd_inner_microstep: 298.35 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 940 [Rank 0] time (ms) | fwd_microstep: 177.22 | bwd_microstep: 279.70 | bwd_inner_microstep: 279.53 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.25 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 185.33 | bwd_microstep: 272.66 | bwd_inner_microstep: 272.47 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 173.99 | bwd_microstep: 269.44 | bwd_inner_microstep: 269.21 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 168.99 | bwd_microstep: 233.81 | bwd_inner_microstep: 233.54 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 176.35 | bwd_microstep: 260.18 | bwd_inner_microstep: 260.00 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.01 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 176.08 | bwd_microstep: 325.06 | bwd_inner_microstep: 309.97 | bwd_allreduce_microstep: 14.99 | step_microstep: 22.64 [Rank 0] time (ms) | fwd: 1400.95 | bwd: 2197.54 | bwd_inner: 2181.03 | bwd_allreduce: 15.70 | step: 23.58 10%|▉ | 273/2865 [1:52:08<37:10:22, 51.63s/it] {'loss': 1.131, 'learning_rate': 3.9554766438475526e-05, 'epoch': 0.29} 10%|▉ | 273/2865 [1:52:08<37:10:22, 51.63s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 179.15 | bwd_microstep: 273.07 | bwd_inner_microstep: 272.80 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 181.31 | bwd_microstep: 303.02 | bwd_inner_microstep: 302.81 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 178.72 | bwd_microstep: 277.23 | bwd_inner_microstep: 277.01 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 178.54 | bwd_microstep: 277.55 | bwd_inner_microstep: 277.18 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 179.42 | bwd_microstep: 271.30 | bwd_inner_microstep: 270.93 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 164.57 | bwd_microstep: 233.72 | bwd_inner_microstep: 233.57 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 166.04 | bwd_microstep: 273.72 | bwd_inner_microstep: 273.53 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 167.49 | bwd_microstep: 279.45 | bwd_inner_microstep: 268.98 | bwd_allreduce_microstep: 10.38 | step_microstep: 21.17 [Rank 0] time (ms) | fwd: 1395.11 | bwd: 2189.20 | bwd_inner: 2177.22 | bwd_allreduce: 11.12 | step: 22.07 10%|▉ | 274/2865 [1:52:12<26:49:37, 37.27s/it] {'loss': 0.8823, 'learning_rate': 3.955000983494028e-05, 'epoch': 0.29} 10%|▉ | 274/2865 [1:52:12<26:49:37, 37.27s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 165.70 | bwd_microstep: 265.00 | bwd_inner_microstep: 264.71 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 175.90 | bwd_microstep: 267.12 | bwd_inner_microstep: 266.83 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 186.11 | bwd_microstep: 253.87 | bwd_inner_microstep: 253.63 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 166.33 | bwd_microstep: 232.30 | bwd_inner_microstep: 232.13 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 174.08 | bwd_microstep: 259.96 | bwd_inner_microstep: 259.58 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 165.85 | bwd_microstep: 233.05 | bwd_inner_microstep: 232.84 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 169.34 | bwd_microstep: 231.47 | bwd_inner_microstep: 231.34 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.95 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 165.65 | bwd_microstep: 270.84 | bwd_inner_microstep: 258.05 | bwd_allreduce_microstep: 12.63 | step_microstep: 21.24 [Rank 0] time (ms) | fwd: 1368.82 | bwd: 2013.75 | bwd_inner: 1999.57 | bwd_allreduce: 13.33 | step: 22.14 10%|▉ | 275/2865 [1:52:16<19:32:30, 27.16s/it] {'loss': 0.9702, 'learning_rate': 3.9545228246966976e-05, 'epoch': 0.29} 10%|▉ | 275/2865 [1:52:16<19:32:30, 27.16s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 162.19 | bwd_microstep: 239.90 | bwd_inner_microstep: 239.60 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 166.47 | bwd_microstep: 258.68 | bwd_inner_microstep: 258.52 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 185.79 | bwd_microstep: 285.32 | bwd_inner_microstep: 285.15 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 178.53 | bwd_microstep: 279.64 | bwd_inner_microstep: 279.35 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 180.60 | bwd_microstep: 237.68 | bwd_inner_microstep: 237.55 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 148.92 | bwd_microstep: 207.49 | bwd_inner_microstep: 207.25 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 171.25 | bwd_microstep: 250.31 | bwd_inner_microstep: 249.95 | bwd_allreduce_microstep: 0.20 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.06 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 170.06 | bwd_microstep: 326.91 | bwd_inner_microstep: 311.51 | bwd_allreduce_microstep: 15.22 | step_microstep: 22.65 [Rank 0] time (ms) | fwd: 1363.64 | bwd: 2086.05 | bwd_inner: 2069.20 | bwd_allreduce: 15.93 | step: 23.41 10%|▉ | 276/2865 [1:52:19<14:27:52, 20.11s/it] {'loss': 1.0037, 'learning_rate': 3.9540421680666375e-05, 'epoch': 0.29} 10%|▉ | 276/2865 [1:52:19<14:27:52, 20.11s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 947 [Rank 0] time (ms) | fwd_microstep: 164.51 | bwd_microstep: 241.23 | bwd_inner_microstep: 241.09 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 169.22 | bwd_microstep: 286.34 | bwd_inner_microstep: 286.18 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 181.18 | bwd_microstep: 277.95 | bwd_inner_microstep: 277.41 | bwd_allreduce_microstep: 0.22 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 177.00 | bwd_microstep: 278.43 | bwd_inner_microstep: 278.03 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.24 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 197.49 | bwd_microstep: 295.76 | bwd_inner_microstep: 295.49 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 178.24 | bwd_microstep: 298.17 | bwd_inner_microstep: 297.93 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 184.50 | bwd_microstep: 273.35 | bwd_inner_microstep: 273.14 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.04 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 176.00 | bwd_microstep: 338.33 | bwd_inner_microstep: 322.00 | bwd_allreduce_microstep: 16.21 | step_microstep: 22.16 [Rank 0] time (ms) | fwd: 1427.99 | bwd: 2289.71 | bwd_inner: 2271.68 | bwd_allreduce: 17.06 | step: 23.15 10%|▉ | 277/2865 [1:52:23<10:58:19, 15.26s/it] {'loss': 0.9346, 'learning_rate': 3.953559014218114e-05, 'epoch': 0.29} 10%|▉ | 277/2865 [1:52:23<10:58:19, 15.26s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 945 [Rank 0] time (ms) | fwd_microstep: 174.29 | bwd_microstep: 283.78 | bwd_inner_microstep: 283.46 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 180.11 | bwd_microstep: 284.36 | bwd_inner_microstep: 284.19 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 176.56 | bwd_microstep: 283.50 | bwd_inner_microstep: 283.24 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 174.06 | bwd_microstep: 234.29 | bwd_inner_microstep: 234.15 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 164.04 | bwd_microstep: 229.02 | bwd_inner_microstep: 228.89 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.83 | bwd_microstep: 234.99 | bwd_inner_microstep: 234.81 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 164.19 | bwd_microstep: 229.64 | bwd_inner_microstep: 229.23 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.94 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 167.08 | bwd_microstep: 258.83 | bwd_inner_microstep: 245.72 | bwd_allreduce_microstep: 13.00 | step_microstep: 23.21 [Rank 0] time (ms) | fwd: 1362.01 | bwd: 2038.56 | bwd_inner: 2024.00 | bwd_allreduce: 13.76 | step: 24.05 10%|▉ | 278/2865 [1:52:27<8:27:06, 11.76s/it] {'loss': 0.8902, 'learning_rate': 3.9530733637685876e-05, 'epoch': 0.29} 10%|▉ | 278/2865 [1:52:27<8:27:06, 11.76s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 162.84 | bwd_microstep: 231.24 | bwd_inner_microstep: 230.98 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 164.26 | bwd_microstep: 242.65 | bwd_inner_microstep: 242.47 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 163.65 | bwd_microstep: 266.91 | bwd_inner_microstep: 266.72 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 162.14 | bwd_microstep: 250.12 | bwd_inner_microstep: 249.99 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 166.63 | bwd_microstep: 233.37 | bwd_inner_microstep: 233.24 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 163.59 | bwd_microstep: 232.24 | bwd_inner_microstep: 232.16 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 166.13 | bwd_microstep: 248.87 | bwd_inner_microstep: 248.76 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.05 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 169.09 | bwd_microstep: 331.99 | bwd_inner_microstep: 320.33 | bwd_allreduce_microstep: 11.45 | step_microstep: 21.97 [Rank 0] time (ms) | fwd: 1318.17 | bwd: 2037.51 | bwd_inner: 2024.91 | bwd_allreduce: 11.96 | step: 22.63 10%|▉ | 279/2865 [1:52:30<6:40:33, 9.29s/it] {'loss': 0.8125, 'learning_rate': 3.952585217338707e-05, 'epoch': 0.29} 10%|▉ | 279/2865 [1:52:30<6:40:33, 9.29s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 178.07 | bwd_microstep: 285.26 | bwd_inner_microstep: 285.10 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 191.57 | bwd_microstep: 283.50 | bwd_inner_microstep: 283.29 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 203.76 | bwd_microstep: 290.78 | bwd_inner_microstep: 290.48 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 178.00 | bwd_microstep: 265.10 | bwd_inner_microstep: 264.88 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 196.59 | bwd_microstep: 306.19 | bwd_inner_microstep: 305.99 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 185.82 | bwd_microstep: 259.13 | bwd_inner_microstep: 258.77 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 174.36 | bwd_microstep: 315.92 | bwd_inner_microstep: 315.57 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.32 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 155.00 | bwd_microstep: 251.00 | bwd_inner_microstep: 235.53 | bwd_allreduce_microstep: 15.28 | step_microstep: 23.37 [Rank 0] time (ms) | fwd: 1462.99 | bwd: 2257.02 | bwd_inner: 2240.10 | bwd_allreduce: 15.95 | step: 24.07 10%|▉ | 280/2865 [1:52:34<5:33:10, 7.73s/it] {'loss': 0.9895, 'learning_rate': 3.952094575552312e-05, 'epoch': 0.29} 10%|▉ | 280/2865 [1:52:34<5:33:10, 7.73s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 178.72 | bwd_microstep: 256.03 | bwd_inner_microstep: 255.87 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 166.94 | bwd_microstep: 238.90 | bwd_inner_microstep: 238.61 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 184.36 | bwd_microstep: 275.97 | bwd_inner_microstep: 275.82 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 177.71 | bwd_microstep: 299.65 | bwd_inner_microstep: 299.43 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 174.95 | bwd_microstep: 245.54 | bwd_inner_microstep: 245.37 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 167.54 | bwd_microstep: 243.57 | bwd_inner_microstep: 243.19 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 164.32 | bwd_microstep: 270.25 | bwd_inner_microstep: 269.97 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.93 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 162.98 | bwd_microstep: 280.17 | bwd_inner_microstep: 266.42 | bwd_allreduce_microstep: 13.58 | step_microstep: 21.71 [Rank 0] time (ms) | fwd: 1377.41 | bwd: 2110.20 | bwd_inner: 2095.03 | bwd_allreduce: 14.29 | step: 22.60 10%|▉ | 281/2865 [1:52:38<4:40:38, 6.52s/it] {'loss': 0.8237, 'learning_rate': 3.9516014390364306e-05, 'epoch': 0.29} 10%|▉ | 281/2865 [1:52:38<4:40:38, 6.52s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 163.60 | bwd_microstep: 266.11 | bwd_inner_microstep: 265.84 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 169.93 | bwd_microstep: 250.78 | bwd_inner_microstep: 250.40 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.27 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 174.09 | bwd_microstep: 285.95 | bwd_inner_microstep: 285.76 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 179.75 | bwd_microstep: 317.54 | bwd_inner_microstep: 317.35 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 187.80 | bwd_microstep: 289.38 | bwd_inner_microstep: 289.19 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 163.19 | bwd_microstep: 254.28 | bwd_inner_microstep: 254.03 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 191.36 | bwd_microstep: 242.75 | bwd_inner_microstep: 242.61 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.00 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 165.99 | bwd_microstep: 271.78 | bwd_inner_microstep: 261.48 | bwd_allreduce_microstep: 10.20 | step_microstep: 21.98 [Rank 0] time (ms) | fwd: 1395.60 | bwd: 2178.71 | bwd_inner: 2166.99 | bwd_allreduce: 10.90 | step: 22.89 10%|▉ | 282/2865 [1:52:42<4:05:00, 5.69s/it] {'loss': 0.877, 'learning_rate': 3.951105808421279e-05, 'epoch': 0.3} 10%|▉ | 282/2865 [1:52:42<4:05:00, 5.69s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 169.29 | bwd_microstep: 233.56 | bwd_inner_microstep: 233.15 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 163.00 | bwd_microstep: 243.85 | bwd_inner_microstep: 243.71 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 175.33 | bwd_microstep: 270.66 | bwd_inner_microstep: 270.51 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 164.14 | bwd_microstep: 247.11 | bwd_inner_microstep: 246.79 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 163.76 | bwd_microstep: 251.16 | bwd_inner_microstep: 250.84 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 168.89 | bwd_microstep: 260.72 | bwd_inner_microstep: 260.57 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 163.35 | bwd_microstep: 247.53 | bwd_inner_microstep: 247.38 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.95 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 172.51 | bwd_microstep: 314.45 | bwd_inner_microstep: 304.14 | bwd_allreduce_microstep: 10.20 | step_microstep: 21.20 [Rank 0] time (ms) | fwd: 1340.15 | bwd: 2069.13 | bwd_inner: 2057.48 | bwd_allreduce: 10.88 | step: 21.82 10%|▉ | 283/2865 [1:52:45<3:37:37, 5.06s/it] {'loss': 0.9399, 'learning_rate': 3.9506076843402603e-05, 'epoch': 0.3} 10%|▉ | 283/2865 [1:52:45<3:37:37, 5.06s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 170.16 | bwd_microstep: 230.28 | bwd_inner_microstep: 229.88 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 171.14 | bwd_microstep: 265.86 | bwd_inner_microstep: 265.56 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 171.52 | bwd_microstep: 272.42 | bwd_inner_microstep: 272.26 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 175.43 | bwd_microstep: 324.56 | bwd_inner_microstep: 324.22 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 177.50 | bwd_microstep: 328.64 | bwd_inner_microstep: 328.45 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 179.95 | bwd_microstep: 293.92 | bwd_inner_microstep: 293.72 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 187.91 | bwd_microstep: 278.61 | bwd_inner_microstep: 278.44 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.22 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 185.97 | bwd_microstep: 346.52 | bwd_inner_microstep: 332.29 | bwd_allreduce_microstep: 14.08 | step_microstep: 25.29 [Rank 0] time (ms) | fwd: 1419.45 | bwd: 2340.95 | bwd_inner: 2325.25 | bwd_allreduce: 14.81 | step: 26.07 10%|▉ | 284/2865 [1:52:49<3:23:16, 4.73s/it] {'loss': 0.9049, 'learning_rate': 3.950107067429967e-05, 'epoch': 0.3} 10%|▉ | 284/2865 [1:52:49<3:23:16, 4.73s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 176.69 | bwd_microstep: 247.91 | bwd_inner_microstep: 247.76 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 177.89 | bwd_microstep: 297.79 | bwd_inner_microstep: 297.48 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 170.15 | bwd_microstep: 307.51 | bwd_inner_microstep: 307.35 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 178.27 | bwd_microstep: 283.98 | bwd_inner_microstep: 283.84 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 179.31 | bwd_microstep: 260.32 | bwd_inner_microstep: 260.18 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 164.18 | bwd_microstep: 249.06 | bwd_inner_microstep: 248.92 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 176.17 | bwd_microstep: 269.25 | bwd_inner_microstep: 269.09 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.98 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 164.91 | bwd_microstep: 291.34 | bwd_inner_microstep: 277.53 | bwd_allreduce_microstep: 13.71 | step_microstep: 21.79 [Rank 0] time (ms) | fwd: 1387.44 | bwd: 2207.27 | bwd_inner: 2192.37 | bwd_allreduce: 14.28 | step: 22.38 10%|▉ | 285/2865 [1:52:53<3:11:13, 4.45s/it] {'loss': 0.9688, 'learning_rate': 3.949603958330175e-05, 'epoch': 0.3} 10%|▉ | 285/2865 [1:52:53<3:11:13, 4.45s/it][INFO|trainer.py:3242] 2026-05-14 22:13:36,359 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 22:13:36,359 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 22:13:36,359 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-285 [INFO|configuration_utils.py:473] 2026-05-14 22:18:47,160 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-285/config.json [INFO|configuration_utils.py:594] 2026-05-14 22:18:47,170 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-285/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 22:18:54,282 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-285/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 22:18:54,313 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-285/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 22:18:54,321 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-285/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 22:18:54,326 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-285/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 22:19:06,511 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-210] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 951 [Rank 0] time (ms) | fwd_microstep: 306545.51 | bwd_microstep: 233.10 | bwd_inner_microstep: 232.93 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 193.63 | bwd_microstep: 277.92 | bwd_inner_microstep: 277.50 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.24 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 191.36 | bwd_microstep: 243.58 | bwd_inner_microstep: 243.45 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 162.99 | bwd_microstep: 219.24 | bwd_inner_microstep: 218.85 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.04 | bwd_microstep: 225.40 | bwd_inner_microstep: 225.07 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 165.87 | bwd_microstep: 224.14 | bwd_inner_microstep: 223.77 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.73 | bwd_microstep: 226.24 | bwd_inner_microstep: 225.87 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.02 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 160.89 | bwd_microstep: 256.10 | bwd_inner_microstep: 245.93 | bwd_allreduce_microstep: 10.07 | step_microstep: 21.90 [Rank 0] time (ms) | fwd: 307728.26 | bwd: 1905.94 | bwd_inner: 1893.86 | bwd_allreduce: 10.98 | step: 22.96 10%|▉ | 286/2865 [1:58:27<73:57:26, 103.24s/it] {'loss': 0.6467, 'learning_rate': 3.949098357683843e-05, 'epoch': 0.3} 10%|▉ | 286/2865 [1:58:27<73:57:26, 103.24s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 163.96 | bwd_microstep: 228.27 | bwd_inner_microstep: 228.14 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 164.61 | bwd_microstep: 230.66 | bwd_inner_microstep: 230.52 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 164.91 | bwd_microstep: 228.21 | bwd_inner_microstep: 227.93 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 169.93 | bwd_microstep: 232.86 | bwd_inner_microstep: 232.58 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 181.25 | bwd_microstep: 217.53 | bwd_inner_microstep: 217.06 | bwd_allreduce_microstep: 0.24 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 164.79 | bwd_microstep: 207.59 | bwd_inner_microstep: 207.24 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.51 | bwd_microstep: 215.67 | bwd_inner_microstep: 215.54 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 166.99 | bwd_microstep: 252.86 | bwd_inner_microstep: 242.70 | bwd_allreduce_microstep: 10.08 | step_microstep: 21.26 [Rank 0] time (ms) | fwd: 1337.82 | bwd: 1813.77 | bwd_inner: 1802.11 | bwd_allreduce: 10.75 | step: 22.10 10%|█ | 287/2865 [1:58:30<52:27:56, 73.26s/it] {'loss': 1.1168, 'learning_rate': 3.948590266137119e-05, 'epoch': 0.3} 10%|█ | 287/2865 [1:58:30<52:27:56, 73.26s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 162.22 | bwd_microstep: 197.82 | bwd_inner_microstep: 197.49 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 162.74 | bwd_microstep: 205.85 | bwd_inner_microstep: 205.72 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.12 | bwd_microstep: 219.95 | bwd_inner_microstep: 219.69 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 161.17 | bwd_microstep: 223.96 | bwd_inner_microstep: 223.60 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 176.31 | bwd_microstep: 227.17 | bwd_inner_microstep: 226.92 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 183.26 | bwd_microstep: 236.38 | bwd_inner_microstep: 236.10 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 160.54 | bwd_microstep: 221.39 | bwd_inner_microstep: 221.03 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.04 | bwd_microstep: 283.55 | bwd_inner_microstep: 273.18 | bwd_allreduce_microstep: 10.20 | step_microstep: 21.23 [Rank 0] time (ms) | fwd: 1328.29 | bwd: 1816.24 | bwd_inner: 1804.23 | bwd_allreduce: 10.91 | step: 21.99 10%|█ | 288/2865 [1:58:34<37:25:19, 52.28s/it] {'loss': 0.8498, 'learning_rate': 3.948079684339331e-05, 'epoch': 0.3} 10%|█ | 288/2865 [1:58:34<37:25:19, 52.28s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 161.39 | bwd_microstep: 227.08 | bwd_inner_microstep: 226.73 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.45 | bwd_microstep: 225.26 | bwd_inner_microstep: 224.85 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 167.75 | bwd_microstep: 260.15 | bwd_inner_microstep: 259.88 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 172.47 | bwd_microstep: 301.60 | bwd_inner_microstep: 301.29 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 177.50 | bwd_microstep: 252.35 | bwd_inner_microstep: 252.17 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.94 | bwd_microstep: 232.03 | bwd_inner_microstep: 231.75 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.64 | bwd_microstep: 231.21 | bwd_inner_microstep: 231.09 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 173.08 | bwd_microstep: 273.97 | bwd_inner_microstep: 263.89 | bwd_allreduce_microstep: 9.93 | step_microstep: 21.92 [Rank 0] time (ms) | fwd: 1335.08 | bwd: 2003.81 | bwd_inner: 1992.14 | bwd_allreduce: 10.67 | step: 22.66 10%|█ | 289/2865 [1:58:37<26:56:29, 37.65s/it] {'loss': 0.9051, 'learning_rate': 3.94756661294299e-05, 'epoch': 0.3} 10%|█ | 289/2865 [1:58:37<26:56:29, 37.65s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 163.44 | bwd_microstep: 233.05 | bwd_inner_microstep: 232.65 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 162.30 | bwd_microstep: 263.30 | bwd_inner_microstep: 263.11 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 164.39 | bwd_microstep: 225.22 | bwd_inner_microstep: 224.87 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 166.75 | bwd_microstep: 233.50 | bwd_inner_microstep: 233.18 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 161.48 | bwd_microstep: 231.55 | bwd_inner_microstep: 231.34 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 168.54 | bwd_microstep: 221.32 | bwd_inner_microstep: 220.94 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 162.36 | bwd_microstep: 216.88 | bwd_inner_microstep: 216.52 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.95 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.50 | bwd_microstep: 272.87 | bwd_inner_microstep: 262.23 | bwd_allreduce_microstep: 10.52 | step_microstep: 21.67 [Rank 0] time (ms) | fwd: 1310.63 | bwd: 1897.84 | bwd_inner: 1885.34 | bwd_allreduce: 11.44 | step: 22.44 10%|█ | 290/2865 [1:58:40<19:34:50, 27.38s/it] {'loss': 0.8937, 'learning_rate': 3.947051052603789e-05, 'epoch': 0.3} 10%|█ | 290/2865 [1:58:40<19:34:50, 27.38s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 161.00 | bwd_microstep: 237.16 | bwd_inner_microstep: 236.98 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 163.53 | bwd_microstep: 235.66 | bwd_inner_microstep: 235.27 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 165.06 | bwd_microstep: 225.04 | bwd_inner_microstep: 224.67 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 146.82 | bwd_microstep: 203.47 | bwd_inner_microstep: 203.12 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 161.65 | bwd_microstep: 221.87 | bwd_inner_microstep: 221.51 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.09 | bwd_microstep: 225.21 | bwd_inner_microstep: 224.81 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 169.37 | bwd_microstep: 217.83 | bwd_inner_microstep: 217.35 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.26 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 160.87 | bwd_microstep: 251.39 | bwd_inner_microstep: 241.45 | bwd_allreduce_microstep: 9.76 | step_microstep: 20.98 [Rank 0] time (ms) | fwd: 1289.27 | bwd: 1817.86 | bwd_inner: 1805.79 | bwd_allreduce: 10.77 | step: 21.95 10%|█ | 291/2865 [1:58:44<14:24:24, 20.15s/it] {'loss': 1.1148, 'learning_rate': 3.946533003980601e-05, 'epoch': 0.3} 10%|█ | 291/2865 [1:58:44<14:24:24, 20.15s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 161.57 | bwd_microstep: 218.48 | bwd_inner_microstep: 218.12 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 170.28 | bwd_microstep: 219.79 | bwd_inner_microstep: 219.44 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 163.26 | bwd_microstep: 225.99 | bwd_inner_microstep: 225.63 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 169.28 | bwd_microstep: 264.54 | bwd_inner_microstep: 264.15 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 179.62 | bwd_microstep: 252.93 | bwd_inner_microstep: 252.52 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.87 | bwd_microstep: 220.53 | bwd_inner_microstep: 220.21 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 167.99 | bwd_microstep: 244.51 | bwd_inner_microstep: 244.37 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.94 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.23 | bwd_microstep: 264.84 | bwd_inner_microstep: 253.77 | bwd_allreduce_microstep: 10.90 | step_microstep: 21.13 [Rank 0] time (ms) | fwd: 1333.97 | bwd: 1911.82 | bwd_inner: 1898.71 | bwd_allreduce: 11.84 | step: 22.07 10%|█ | 292/2865 [1:58:47<10:48:53, 15.13s/it] {'loss': 1.139, 'learning_rate': 3.9460124677354797e-05, 'epoch': 0.31} 10%|█ | 292/2865 [1:58:47<10:48:53, 15.13s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 162.03 | bwd_microstep: 234.77 | bwd_inner_microstep: 234.62 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 166.44 | bwd_microstep: 228.92 | bwd_inner_microstep: 228.37 | bwd_allreduce_microstep: 0.24 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 162.22 | bwd_microstep: 232.07 | bwd_inner_microstep: 231.73 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.27 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.60 | bwd_microstep: 258.92 | bwd_inner_microstep: 258.74 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 158.01 | bwd_microstep: 254.35 | bwd_inner_microstep: 253.92 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.31 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 202.29 | bwd_microstep: 265.89 | bwd_inner_microstep: 265.47 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 174.37 | bwd_microstep: 272.51 | bwd_inner_microstep: 272.35 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.12 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 184.43 | bwd_microstep: 337.16 | bwd_inner_microstep: 325.80 | bwd_allreduce_microstep: 11.16 | step_microstep: 22.45 [Rank 0] time (ms) | fwd: 1373.23 | bwd: 2084.81 | bwd_inner: 2071.47 | bwd_allreduce: 12.15 | step: 23.79 10%|█ | 293/2865 [1:58:51<8:21:27, 11.70s/it] {'loss': 0.9265, 'learning_rate': 3.945489444533658e-05, 'epoch': 0.31} 10%|█ | 293/2865 [1:58:51<8:21:27, 11.70s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 185.58 | bwd_microstep: 290.48 | bwd_inner_microstep: 290.00 | bwd_allreduce_microstep: 0.21 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 180.31 | bwd_microstep: 305.77 | bwd_inner_microstep: 305.46 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 177.20 | bwd_microstep: 284.14 | bwd_inner_microstep: 283.85 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 180.14 | bwd_microstep: 278.28 | bwd_inner_microstep: 277.79 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.24 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 178.96 | bwd_microstep: 293.96 | bwd_inner_microstep: 293.60 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.09 | bwd_microstep: 289.49 | bwd_inner_microstep: 289.30 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 176.69 | bwd_microstep: 281.77 | bwd_inner_microstep: 281.43 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.18 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 195.63 | bwd_microstep: 326.55 | bwd_inner_microstep: 315.16 | bwd_allreduce_microstep: 11.29 | step_microstep: 23.88 [Rank 0] time (ms) | fwd: 1434.45 | bwd: 2350.64 | bwd_inner: 2337.17 | bwd_allreduce: 12.30 | step: 24.98 10%|█ | 294/2865 [1:58:55<6:42:26, 9.39s/it] {'loss': 0.9947, 'learning_rate': 3.944963935043548e-05, 'epoch': 0.31} 10%|█ | 294/2865 [1:58:55<6:42:26, 9.39s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 195.09 | bwd_microstep: 272.20 | bwd_inner_microstep: 272.04 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 218.56 | bwd_microstep: 321.83 | bwd_inner_microstep: 321.46 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 178.24 | bwd_microstep: 333.54 | bwd_inner_microstep: 333.38 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 177.28 | bwd_microstep: 265.04 | bwd_inner_microstep: 264.89 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 172.95 | bwd_microstep: 295.32 | bwd_inner_microstep: 295.17 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 171.72 | bwd_microstep: 309.49 | bwd_inner_microstep: 309.18 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 213.41 | bwd_microstep: 280.00 | bwd_inner_microstep: 279.83 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.01 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 181.76 | bwd_microstep: 364.91 | bwd_inner_microstep: 350.99 | bwd_allreduce_microstep: 13.73 | step_microstep: 22.34 [Rank 0] time (ms) | fwd: 1508.86 | bwd: 2442.42 | bwd_inner: 2427.29 | bwd_allreduce: 14.31 | step: 23.06 10%|█ | 295/2865 [1:58:59<5:35:07, 7.82s/it] {'loss': 0.9887, 'learning_rate': 3.944435939936736e-05, 'epoch': 0.31} 10%|█ | 295/2865 [1:58:59<5:35:07, 7.82s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 180.35 | bwd_microstep: 283.11 | bwd_inner_microstep: 282.68 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 211.45 | bwd_microstep: 279.09 | bwd_inner_microstep: 278.80 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 210.00 | bwd_microstep: 274.61 | bwd_inner_microstep: 274.34 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 184.04 | bwd_microstep: 276.78 | bwd_inner_microstep: 276.47 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 187.36 | bwd_microstep: 301.98 | bwd_inner_microstep: 301.69 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 182.96 | bwd_microstep: 315.49 | bwd_inner_microstep: 314.97 | bwd_allreduce_microstep: 0.21 | step_microstep: 0.25 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 173.12 | bwd_microstep: 308.38 | bwd_inner_microstep: 308.07 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.06 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 174.82 | bwd_microstep: 358.18 | bwd_inner_microstep: 346.34 | bwd_allreduce_microstep: 11.66 | step_microstep: 22.12 [Rank 0] time (ms) | fwd: 1503.89 | bwd: 2397.77 | bwd_inner: 2384.03 | bwd_allreduce: 12.49 | step: 23.06 10%|█ | 296/2865 [1:59:03<4:47:39, 6.72s/it] {'loss': 0.8973, 'learning_rate': 3.943905459887989e-05, 'epoch': 0.31} 10%|█ | 296/2865 [1:59:03<4:47:39, 6.72s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 191.04 | bwd_microstep: 274.52 | bwd_inner_microstep: 274.18 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 171.03 | bwd_microstep: 262.28 | bwd_inner_microstep: 261.88 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 181.46 | bwd_microstep: 265.38 | bwd_inner_microstep: 264.90 | bwd_allreduce_microstep: 0.28 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 183.83 | bwd_microstep: 267.34 | bwd_inner_microstep: 267.00 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 170.52 | bwd_microstep: 269.09 | bwd_inner_microstep: 268.82 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 172.19 | bwd_microstep: 289.24 | bwd_inner_microstep: 288.92 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 193.87 | bwd_microstep: 285.01 | bwd_inner_microstep: 284.68 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.99 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 156.27 | bwd_microstep: 311.30 | bwd_inner_microstep: 300.01 | bwd_allreduce_microstep: 11.19 | step_microstep: 21.98 [Rank 0] time (ms) | fwd: 1420.06 | bwd: 2224.29 | bwd_inner: 2210.91 | bwd_allreduce: 12.24 | step: 22.76 10%|█ | 297/2865 [1:59:07<4:10:48, 5.86s/it] {'loss': 0.961, 'learning_rate': 3.943372495575245e-05, 'epoch': 0.31} 10%|█ | 297/2865 [1:59:07<4:10:48, 5.86s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 176.31 | bwd_microstep: 293.79 | bwd_inner_microstep: 293.45 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 177.29 | bwd_microstep: 316.69 | bwd_inner_microstep: 316.51 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 183.92 | bwd_microstep: 304.03 | bwd_inner_microstep: 303.85 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 178.49 | bwd_microstep: 351.53 | bwd_inner_microstep: 351.13 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 195.62 | bwd_microstep: 315.94 | bwd_inner_microstep: 315.64 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 258.44 | bwd_microstep: 261.05 | bwd_inner_microstep: 260.64 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 180.50 | bwd_microstep: 291.94 | bwd_inner_microstep: 291.78 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.96 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 165.33 | bwd_microstep: 314.37 | bwd_inner_microstep: 303.32 | bwd_allreduce_microstep: 10.90 | step_microstep: 21.43 [Rank 0] time (ms) | fwd: 1515.73 | bwd: 2449.49 | bwd_inner: 2436.78 | bwd_allreduce: 11.68 | step: 22.35 10%|█ | 298/2865 [1:59:11<3:49:16, 5.36s/it] {'loss': 1.0102, 'learning_rate': 3.942837047679622e-05, 'epoch': 0.31} 10%|█ | 298/2865 [1:59:11<3:49:16, 5.36s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 179.57 | bwd_microstep: 279.85 | bwd_inner_microstep: 279.54 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 179.93 | bwd_microstep: 280.54 | bwd_inner_microstep: 280.30 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 182.00 | bwd_microstep: 303.90 | bwd_inner_microstep: 303.72 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 206.72 | bwd_microstep: 282.83 | bwd_inner_microstep: 282.64 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 181.75 | bwd_microstep: 293.58 | bwd_inner_microstep: 293.41 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 191.62 | bwd_microstep: 279.53 | bwd_inner_microstep: 279.38 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 177.38 | bwd_microstep: 290.85 | bwd_inner_microstep: 290.45 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.13 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 181.19 | bwd_microstep: 328.24 | bwd_inner_microstep: 315.37 | bwd_allreduce_microstep: 12.77 | step_microstep: 23.52 [Rank 0] time (ms) | fwd: 1480.00 | bwd: 2339.43 | bwd_inner: 2325.20 | bwd_allreduce: 13.40 | step: 24.28 10%|█ | 299/2865 [1:59:15<3:32:25, 4.97s/it] {'loss': 0.9487, 'learning_rate': 3.942299116885409e-05, 'epoch': 0.31} 10%|█ | 299/2865 [1:59:15<3:32:25, 4.97s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 187.02 | bwd_microstep: 294.73 | bwd_inner_microstep: 294.31 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 179.35 | bwd_microstep: 300.79 | bwd_inner_microstep: 300.56 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 190.35 | bwd_microstep: 284.52 | bwd_inner_microstep: 284.13 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 183.06 | bwd_microstep: 294.90 | bwd_inner_microstep: 294.74 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 183.46 | bwd_microstep: 293.22 | bwd_inner_microstep: 292.91 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 176.88 | bwd_microstep: 323.02 | bwd_inner_microstep: 322.82 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 196.96 | bwd_microstep: 297.78 | bwd_inner_microstep: 297.60 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.98 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 189.09 | bwd_microstep: 335.12 | bwd_inner_microstep: 324.15 | bwd_allreduce_microstep: 10.87 | step_microstep: 21.49 [Rank 0] time (ms) | fwd: 1486.03 | bwd: 2424.25 | bwd_inner: 2411.61 | bwd_allreduce: 11.66 | step: 22.25 10%|█ | 300/2865 [1:59:19<3:21:20, 4.71s/it] {'loss': 1.0122, 'learning_rate': 3.9417587038800666e-05, 'epoch': 0.31} 10%|█ | 300/2865 [1:59:19<3:21:20, 4.71s/it][INFO|trainer.py:3242] 2026-05-14 22:20:02,587 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 22:20:02,587 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 22:20:02,587 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-300 [INFO|configuration_utils.py:473] 2026-05-14 22:25:15,138 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-300/config.json [INFO|configuration_utils.py:594] 2026-05-14 22:25:15,148 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-300/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 22:25:21,317 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-300/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 22:25:21,327 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-300/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 22:25:21,333 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-300/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 22:25:21,335 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-300/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 22:25:29,070 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-225] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 307680.96 | bwd_microstep: 226.67 | bwd_inner_microstep: 226.51 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 169.60 | bwd_microstep: 247.48 | bwd_inner_microstep: 247.15 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.25 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 168.36 | bwd_microstep: 232.99 | bwd_inner_microstep: 232.84 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 172.16 | bwd_microstep: 229.70 | bwd_inner_microstep: 229.29 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 165.38 | bwd_microstep: 232.84 | bwd_inner_microstep: 232.74 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 166.80 | bwd_microstep: 264.53 | bwd_inner_microstep: 264.27 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 166.08 | bwd_microstep: 233.40 | bwd_inner_microstep: 233.24 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.02 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 164.71 | bwd_microstep: 280.19 | bwd_inner_microstep: 268.85 | bwd_allreduce_microstep: 11.21 | step_microstep: 21.80 [Rank 0] time (ms) | fwd: 308840.02 | bwd: 1947.94 | bwd_inner: 1935.17 | bwd_allreduce: 11.92 | step: 22.65 11%|█ | 301/2865 [2:04:50<72:53:12, 102.34s/it] {'loss': 0.8358, 'learning_rate': 3.941215809354231e-05, 'epoch': 0.31} 11%|█ | 301/2865 [2:04:50<72:53:12, 102.34s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 940 [Rank 0] time (ms) | fwd_microstep: 166.48 | bwd_microstep: 228.94 | bwd_inner_microstep: 228.56 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 167.33 | bwd_microstep: 240.98 | bwd_inner_microstep: 240.78 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 173.71 | bwd_microstep: 242.34 | bwd_inner_microstep: 242.05 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 164.82 | bwd_microstep: 229.66 | bwd_inner_microstep: 229.51 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 164.90 | bwd_microstep: 243.31 | bwd_inner_microstep: 243.19 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 162.38 | bwd_microstep: 233.99 | bwd_inner_microstep: 233.85 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 161.43 | bwd_microstep: 228.81 | bwd_inner_microstep: 228.54 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.31 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 163.57 | bwd_microstep: 262.99 | bwd_inner_microstep: 250.67 | bwd_allreduce_microstep: 12.16 | step_microstep: 23.21 [Rank 0] time (ms) | fwd: 1324.49 | bwd: 1911.14 | bwd_inner: 1897.56 | bwd_allreduce: 12.72 | step: 23.83 11%|█ | 302/2865 [2:04:53<51:43:51, 72.66s/it] {'loss': 0.8969, 'learning_rate': 3.9406704340017085e-05, 'epoch': 0.32} 11%|█ | 302/2865 [2:04:53<51:43:51, 72.66s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 165.64 | bwd_microstep: 261.71 | bwd_inner_microstep: 261.53 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 165.95 | bwd_microstep: 247.36 | bwd_inner_microstep: 247.21 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 163.35 | bwd_microstep: 220.11 | bwd_inner_microstep: 219.94 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 162.18 | bwd_microstep: 233.81 | bwd_inner_microstep: 233.53 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 161.84 | bwd_microstep: 223.09 | bwd_inner_microstep: 222.96 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.20 | bwd_microstep: 236.38 | bwd_inner_microstep: 236.23 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 162.44 | bwd_microstep: 229.09 | bwd_inner_microstep: 228.96 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.37 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 163.02 | bwd_microstep: 272.37 | bwd_inner_microstep: 262.51 | bwd_allreduce_microstep: 9.76 | step_microstep: 26.73 [Rank 0] time (ms) | fwd: 1307.51 | bwd: 1924.03 | bwd_inner: 1913.09 | bwd_allreduce: 10.29 | step: 27.30 11%|█ | 303/2865 [2:04:56<36:55:29, 51.89s/it] {'loss': 0.7069, 'learning_rate': 3.940122578519475e-05, 'epoch': 0.32} 11%|█ | 303/2865 [2:04:56<36:55:29, 51.89s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 165.98 | bwd_microstep: 227.62 | bwd_inner_microstep: 227.36 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 165.05 | bwd_microstep: 241.39 | bwd_inner_microstep: 241.00 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 166.91 | bwd_microstep: 239.01 | bwd_inner_microstep: 238.89 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 164.86 | bwd_microstep: 251.09 | bwd_inner_microstep: 250.96 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 165.81 | bwd_microstep: 240.22 | bwd_inner_microstep: 240.00 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 186.75 | bwd_microstep: 232.57 | bwd_inner_microstep: 232.25 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 168.12 | bwd_microstep: 249.31 | bwd_inner_microstep: 249.13 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.95 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 178.85 | bwd_microstep: 279.89 | bwd_inner_microstep: 269.90 | bwd_allreduce_microstep: 9.90 | step_microstep: 22.46 [Rank 0] time (ms) | fwd: 1362.19 | bwd: 1961.23 | bwd_inner: 1949.85 | bwd_allreduce: 10.58 | step: 23.20 11%|█ | 304/2865 [2:05:00<26:35:11, 37.37s/it] {'loss': 0.9926, 'learning_rate': 3.939572243607676e-05, 'epoch': 0.32} 11%|█ | 304/2865 [2:05:00<26:35:11, 37.37s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 166.23 | bwd_microstep: 246.27 | bwd_inner_microstep: 245.91 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 167.81 | bwd_microstep: 237.46 | bwd_inner_microstep: 237.33 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 171.26 | bwd_microstep: 235.75 | bwd_inner_microstep: 235.56 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 172.28 | bwd_microstep: 331.36 | bwd_inner_microstep: 331.04 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.95 | bwd_microstep: 255.11 | bwd_inner_microstep: 254.87 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 166.12 | bwd_microstep: 258.20 | bwd_inner_microstep: 257.98 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 203.88 | bwd_microstep: 254.78 | bwd_inner_microstep: 254.56 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.28 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 170.63 | bwd_microstep: 373.22 | bwd_inner_microstep: 359.90 | bwd_allreduce_microstep: 13.23 | step_microstep: 22.72 [Rank 0] time (ms) | fwd: 1382.04 | bwd: 2192.30 | bwd_inner: 2177.47 | bwd_allreduce: 13.95 | step: 23.54 11%|█ | 305/2865 [2:05:04<19:24:20, 27.29s/it] {'loss': 0.776, 'learning_rate': 3.9390194299696274e-05, 'epoch': 0.32} 11%|█ | 305/2865 [2:05:04<19:24:20, 27.29s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 167.71 | bwd_microstep: 244.27 | bwd_inner_microstep: 244.10 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 180.22 | bwd_microstep: 241.34 | bwd_inner_microstep: 241.12 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 167.35 | bwd_microstep: 236.64 | bwd_inner_microstep: 236.39 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 171.05 | bwd_microstep: 275.77 | bwd_inner_microstep: 275.63 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 168.21 | bwd_microstep: 255.03 | bwd_inner_microstep: 254.84 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 180.44 | bwd_microstep: 242.78 | bwd_inner_microstep: 242.51 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 169.47 | bwd_microstep: 232.89 | bwd_inner_microstep: 232.76 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 169.86 | bwd_microstep: 282.90 | bwd_inner_microstep: 271.70 | bwd_allreduce_microstep: 11.10 | step_microstep: 21.33 [Rank 0] time (ms) | fwd: 1374.18 | bwd: 2011.74 | bwd_inner: 1999.31 | bwd_allreduce: 11.71 | step: 22.00 11%|█ | 306/2865 [2:05:07<14:20:27, 20.17s/it] {'loss': 0.7522, 'learning_rate': 3.938464138311811e-05, 'epoch': 0.32} 11%|█ | 306/2865 [2:05:07<14:20:27, 20.17s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 166.42 | bwd_microstep: 237.36 | bwd_inner_microstep: 237.20 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 168.37 | bwd_microstep: 284.53 | bwd_inner_microstep: 284.37 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 166.40 | bwd_microstep: 243.40 | bwd_inner_microstep: 243.19 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 171.70 | bwd_microstep: 271.15 | bwd_inner_microstep: 270.81 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 175.65 | bwd_microstep: 259.19 | bwd_inner_microstep: 258.95 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 173.76 | bwd_microstep: 266.41 | bwd_inner_microstep: 266.25 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 151.33 | bwd_microstep: 228.48 | bwd_inner_microstep: 228.25 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.96 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 179.73 | bwd_microstep: 304.64 | bwd_inner_microstep: 294.25 | bwd_allreduce_microstep: 10.30 | step_microstep: 21.81 [Rank 0] time (ms) | fwd: 1353.22 | bwd: 2095.27 | bwd_inner: 2083.58 | bwd_allreduce: 10.95 | step: 22.46 11%|█ | 307/2865 [2:05:11<10:48:37, 15.21s/it] {'loss': 0.8338, 'learning_rate': 3.937906369343876e-05, 'epoch': 0.32} 11%|█ | 307/2865 [2:05:11<10:48:37, 15.21s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 191.69 | bwd_microstep: 256.29 | bwd_inner_microstep: 255.90 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 942 [Rank 0] time (ms) | fwd_microstep: 176.91 | bwd_microstep: 248.90 | bwd_inner_microstep: 248.66 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 941 [Rank 0] time (ms) | fwd_microstep: 169.85 | bwd_microstep: 290.95 | bwd_inner_microstep: 290.72 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 166.31 | bwd_microstep: 236.51 | bwd_inner_microstep: 236.32 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 166.87 | bwd_microstep: 276.76 | bwd_inner_microstep: 276.63 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 165.00 | bwd_microstep: 286.08 | bwd_inner_microstep: 285.96 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 165.32 | bwd_microstep: 242.34 | bwd_inner_microstep: 241.97 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 179.57 | bwd_microstep: 290.47 | bwd_inner_microstep: 279.80 | bwd_allreduce_microstep: 10.55 | step_microstep: 21.28 [Rank 0] time (ms) | fwd: 1381.40 | bwd: 2128.47 | bwd_inner: 2116.31 | bwd_allreduce: 11.22 | step: 22.03 11%|█ | 308/2865 [2:05:15<8:21:08, 11.76s/it] {'loss': 0.8795, 'learning_rate': 3.937346123778638e-05, 'epoch': 0.32} 11%|█ | 308/2865 [2:05:15<8:21:08, 11.76s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 946 [Rank 0] time (ms) | fwd_microstep: 177.38 | bwd_microstep: 236.73 | bwd_inner_microstep: 236.36 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 163.81 | bwd_microstep: 233.54 | bwd_inner_microstep: 233.43 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 166.50 | bwd_microstep: 246.84 | bwd_inner_microstep: 246.73 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 172.49 | bwd_microstep: 245.01 | bwd_inner_microstep: 244.80 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 166.55 | bwd_microstep: 234.03 | bwd_inner_microstep: 233.94 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 168.72 | bwd_microstep: 246.80 | bwd_inner_microstep: 246.64 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 166.74 | bwd_microstep: 254.37 | bwd_inner_microstep: 254.05 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 168.50 | bwd_microstep: 275.92 | bwd_inner_microstep: 261.56 | bwd_allreduce_microstep: 14.25 | step_microstep: 21.54 [Rank 0] time (ms) | fwd: 1350.56 | bwd: 1973.38 | bwd_inner: 1957.76 | bwd_allreduce: 14.86 | step: 22.22 11%|█ | 309/2865 [2:05:18<6:35:24, 9.28s/it] {'loss': 0.9388, 'learning_rate': 3.936783402332076e-05, 'epoch': 0.32} 11%|█ | 309/2865 [2:05:18<6:35:24, 9.28s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 182.67 | bwd_microstep: 240.20 | bwd_inner_microstep: 240.07 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 947 [Rank 0] time (ms) | fwd_microstep: 168.83 | bwd_microstep: 242.89 | bwd_inner_microstep: 242.65 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 165.66 | bwd_microstep: 245.22 | bwd_inner_microstep: 244.91 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 166.64 | bwd_microstep: 255.18 | bwd_inner_microstep: 254.96 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 169.52 | bwd_microstep: 279.95 | bwd_inner_microstep: 279.54 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 178.10 | bwd_microstep: 255.78 | bwd_inner_microstep: 255.60 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 170.21 | bwd_microstep: 237.32 | bwd_inner_microstep: 237.18 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.31 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 179.98 | bwd_microstep: 276.61 | bwd_inner_microstep: 265.50 | bwd_allreduce_microstep: 11.00 | step_microstep: 22.47 [Rank 0] time (ms) | fwd: 1381.48 | bwd: 2033.34 | bwd_inner: 2020.66 | bwd_allreduce: 11.76 | step: 23.35 11%|█ | 310/2865 [2:05:22<5:22:43, 7.58s/it] {'loss': 1.074, 'learning_rate': 3.936218205723336e-05, 'epoch': 0.32} 11%|█ | 310/2865 [2:05:22<5:22:43, 7.58s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 170.98 | bwd_microstep: 233.51 | bwd_inner_microstep: 233.33 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 168.20 | bwd_microstep: 239.37 | bwd_inner_microstep: 238.94 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 170.79 | bwd_microstep: 300.93 | bwd_inner_microstep: 300.77 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 169.23 | bwd_microstep: 234.07 | bwd_inner_microstep: 233.79 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 180.44 | bwd_microstep: 235.02 | bwd_inner_microstep: 234.71 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 176.11 | bwd_microstep: 283.98 | bwd_inner_microstep: 283.86 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 165.88 | bwd_microstep: 254.55 | bwd_inner_microstep: 254.45 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.34 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 166.24 | bwd_microstep: 288.04 | bwd_inner_microstep: 273.29 | bwd_allreduce_microstep: 14.61 | step_microstep: 25.57 [Rank 0] time (ms) | fwd: 1367.75 | bwd: 2069.61 | bwd_inner: 2053.48 | bwd_allreduce: 15.23 | step: 26.23 11%|█ | 311/2865 [2:05:25<4:32:18, 6.40s/it] {'loss': 0.9102, 'learning_rate': 3.9356505346747244e-05, 'epoch': 0.33} 11%|█ | 311/2865 [2:05:25<4:32:18, 6.40s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 168.16 | bwd_microstep: 224.89 | bwd_inner_microstep: 224.49 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 164.31 | bwd_microstep: 240.35 | bwd_inner_microstep: 240.18 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 174.06 | bwd_microstep: 231.24 | bwd_inner_microstep: 231.10 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 163.78 | bwd_microstep: 234.23 | bwd_inner_microstep: 234.08 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 186.24 | bwd_microstep: 238.36 | bwd_inner_microstep: 238.03 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 181.84 | bwd_microstep: 247.70 | bwd_inner_microstep: 247.58 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 165.85 | bwd_microstep: 247.09 | bwd_inner_microstep: 246.95 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.94 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 169.45 | bwd_microstep: 310.91 | bwd_inner_microstep: 299.93 | bwd_allreduce_microstep: 10.85 | step_microstep: 21.91 [Rank 0] time (ms) | fwd: 1373.58 | bwd: 1974.88 | bwd_inner: 1962.68 | bwd_allreduce: 11.44 | step: 22.43 11%|█ | 312/2865 [2:05:29<3:55:29, 5.53s/it] {'loss': 0.742, 'learning_rate': 3.9350803899117116e-05, 'epoch': 0.33} 11%|█ | 312/2865 [2:05:29<3:55:29, 5.53s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 186.48 | bwd_microstep: 281.08 | bwd_inner_microstep: 280.59 | bwd_allreduce_microstep: 0.23 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 190.50 | bwd_microstep: 291.97 | bwd_inner_microstep: 291.67 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 186.36 | bwd_microstep: 284.85 | bwd_inner_microstep: 284.58 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 178.61 | bwd_microstep: 295.04 | bwd_inner_microstep: 294.84 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 182.82 | bwd_microstep: 285.85 | bwd_inner_microstep: 285.22 | bwd_allreduce_microstep: 0.23 | step_microstep: 0.25 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.26 | bwd_microstep: 230.68 | bwd_inner_microstep: 230.42 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 193.65 | bwd_microstep: 223.52 | bwd_inner_microstep: 223.28 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 163.73 | bwd_microstep: 254.12 | bwd_inner_microstep: 244.11 | bwd_allreduce_microstep: 9.89 | step_microstep: 21.39 [Rank 0] time (ms) | fwd: 1442.28 | bwd: 2147.38 | bwd_inner: 2135.23 | bwd_allreduce: 10.90 | step: 22.44 11%|█ | 313/2865 [2:05:33<3:33:11, 5.01s/it] {'loss': 0.7209, 'learning_rate': 3.934507772162928e-05, 'epoch': 0.33} 11%|█ | 313/2865 [2:05:33<3:33:11, 5.01s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 163.19 | bwd_microstep: 226.92 | bwd_inner_microstep: 226.48 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 163.84 | bwd_microstep: 244.39 | bwd_inner_microstep: 244.20 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 166.50 | bwd_microstep: 228.55 | bwd_inner_microstep: 228.44 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 163.59 | bwd_microstep: 226.96 | bwd_inner_microstep: 226.62 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 164.54 | bwd_microstep: 232.25 | bwd_inner_microstep: 232.01 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 163.73 | bwd_microstep: 227.95 | bwd_inner_microstep: 227.76 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 162.92 | bwd_microstep: 230.03 | bwd_inner_microstep: 229.91 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.03 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 165.89 | bwd_microstep: 255.47 | bwd_inner_microstep: 245.56 | bwd_allreduce_microstep: 9.80 | step_microstep: 22.00 [Rank 0] time (ms) | fwd: 1314.11 | bwd: 1872.70 | bwd_inner: 1861.28 | bwd_allreduce: 10.50 | step: 22.92 11%|█ | 314/2865 [2:05:36<3:11:59, 4.52s/it] {'loss': 0.9894, 'learning_rate': 3.9339326821601666e-05, 'epoch': 0.33} 11%|█ | 314/2865 [2:05:36<3:11:59, 4.52s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 161.53 | bwd_microstep: 224.85 | bwd_inner_microstep: 224.67 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 162.77 | bwd_microstep: 222.20 | bwd_inner_microstep: 221.88 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 182.61 | bwd_microstep: 230.29 | bwd_inner_microstep: 230.18 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 161.76 | bwd_microstep: 230.26 | bwd_inner_microstep: 230.13 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 163.50 | bwd_microstep: 230.40 | bwd_inner_microstep: 230.02 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 165.77 | bwd_microstep: 223.62 | bwd_inner_microstep: 223.54 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 164.04 | bwd_microstep: 217.58 | bwd_inner_microstep: 217.43 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.93 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 166.49 | bwd_microstep: 264.70 | bwd_inner_microstep: 250.85 | bwd_allreduce_microstep: 13.72 | step_microstep: 24.00 [Rank 0] time (ms) | fwd: 1328.35 | bwd: 1844.03 | bwd_inner: 1828.98 | bwd_allreduce: 14.30 | step: 24.74 11%|█ | 315/2865 [2:05:39<2:57:04, 4.17s/it] {'loss': 1.1615, 'learning_rate': 3.9333551206383764e-05, 'epoch': 0.33} 11%|█ | 315/2865 [2:05:39<2:57:04, 4.17s/it][INFO|trainer.py:3242] 2026-05-14 22:26:22,496 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 22:26:22,496 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 22:26:22,497 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-315 [INFO|configuration_utils.py:473] 2026-05-14 22:31:31,424 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-315/config.json [INFO|configuration_utils.py:594] 2026-05-14 22:31:31,436 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-315/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 22:31:38,829 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-315/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 22:31:38,853 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-315/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 22:31:38,866 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-315/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 22:31:38,869 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-315/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 22:31:49,338 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-240] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 305335.42 | bwd_microstep: 229.11 | bwd_inner_microstep: 228.67 | bwd_allreduce_microstep: 0.21 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 165.00 | bwd_microstep: 228.52 | bwd_inner_microstep: 228.29 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 160.82 | bwd_microstep: 231.01 | bwd_inner_microstep: 230.88 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 162.00 | bwd_microstep: 227.46 | bwd_inner_microstep: 227.10 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 164.83 | bwd_microstep: 237.90 | bwd_inner_microstep: 237.78 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.90 | bwd_microstep: 222.39 | bwd_inner_microstep: 222.22 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.99 | bwd_microstep: 223.87 | bwd_inner_microstep: 223.47 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.98 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 162.05 | bwd_microstep: 257.58 | bwd_inner_microstep: 247.45 | bwd_allreduce_microstep: 10.03 | step_microstep: 22.26 [Rank 0] time (ms) | fwd: 306460.58 | bwd: 1857.97 | bwd_inner: 1846.24 | bwd_allreduce: 10.83 | step: 22.97 11%|█ | 316/2865 [2:11:10<72:14:31, 102.03s/it] {'loss': 0.8703, 'learning_rate': 3.932775088335669e-05, 'epoch': 0.33} 11%|█ | 316/2865 [2:11:10<72:14:31, 102.03s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 162.55 | bwd_microstep: 220.99 | bwd_inner_microstep: 220.86 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 165.81 | bwd_microstep: 232.56 | bwd_inner_microstep: 232.37 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 160.79 | bwd_microstep: 229.85 | bwd_inner_microstep: 229.73 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 165.42 | bwd_microstep: 223.32 | bwd_inner_microstep: 223.17 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.70 | bwd_microstep: 220.32 | bwd_inner_microstep: 219.96 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 164.97 | bwd_microstep: 226.06 | bwd_inner_microstep: 225.92 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 163.24 | bwd_microstep: 225.56 | bwd_inner_microstep: 225.26 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 165.30 | bwd_microstep: 294.08 | bwd_inner_microstep: 284.09 | bwd_allreduce_microstep: 9.88 | step_microstep: 21.39 [Rank 0] time (ms) | fwd: 1309.68 | bwd: 1872.84 | bwd_inner: 1861.66 | bwd_allreduce: 10.47 | step: 22.04 11%|█ | 317/2865 [2:11:13<51:15:33, 72.42s/it] {'loss': 1.001, 'learning_rate': 3.9321925859933105e-05, 'epoch': 0.33} 11%|█ | 317/2865 [2:11:13<51:15:33, 72.42s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 160.91 | bwd_microstep: 225.03 | bwd_inner_microstep: 224.66 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 162.84 | bwd_microstep: 219.96 | bwd_inner_microstep: 219.56 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 162.79 | bwd_microstep: 222.40 | bwd_inner_microstep: 221.93 | bwd_allreduce_microstep: 0.23 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 165.81 | bwd_microstep: 232.32 | bwd_inner_microstep: 232.00 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 162.44 | bwd_microstep: 224.10 | bwd_inner_microstep: 223.97 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 164.61 | bwd_microstep: 223.29 | bwd_inner_microstep: 222.92 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.19 | bwd_microstep: 224.49 | bwd_inner_microstep: 224.23 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.74 | bwd_microstep: 257.10 | bwd_inner_microstep: 245.58 | bwd_allreduce_microstep: 11.33 | step_microstep: 20.97 [Rank 0] time (ms) | fwd: 1300.23 | bwd: 1828.90 | bwd_inner: 1815.31 | bwd_allreduce: 12.28 | step: 22.00 11%|█ | 318/2865 [2:11:16<36:34:03, 51.69s/it] {'loss': 0.978, 'learning_rate': 3.931607614355726e-05, 'epoch': 0.33} 11%|█ | 318/2865 [2:11:16<36:34:03, 51.69s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 163.91 | bwd_microstep: 222.91 | bwd_inner_microstep: 222.72 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.77 | bwd_microstep: 219.38 | bwd_inner_microstep: 219.29 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 163.66 | bwd_microstep: 222.31 | bwd_inner_microstep: 222.17 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 163.78 | bwd_microstep: 220.80 | bwd_inner_microstep: 220.60 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 161.92 | bwd_microstep: 225.78 | bwd_inner_microstep: 225.20 | bwd_allreduce_microstep: 0.24 | step_microstep: 0.32 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 169.81 | bwd_microstep: 219.15 | bwd_inner_microstep: 218.87 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 156.01 | bwd_microstep: 194.77 | bwd_inner_microstep: 194.54 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 164.92 | bwd_microstep: 253.49 | bwd_inner_microstep: 243.63 | bwd_allreduce_microstep: 9.78 | step_microstep: 21.10 [Rank 0] time (ms) | fwd: 1304.65 | bwd: 1778.75 | bwd_inner: 1767.41 | bwd_allreduce: 10.46 | step: 21.97 11%|█ | 319/2865 [2:11:20<26:16:49, 37.16s/it] {'loss': 0.8977, 'learning_rate': 3.9310201741704933e-05, 'epoch': 0.33} 11%|█ | 319/2865 [2:11:20<26:16:49, 37.16s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 165.41 | bwd_microstep: 237.78 | bwd_inner_microstep: 237.45 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 164.38 | bwd_microstep: 233.21 | bwd_inner_microstep: 232.74 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 165.29 | bwd_microstep: 239.39 | bwd_inner_microstep: 238.89 | bwd_allreduce_microstep: 0.22 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 164.71 | bwd_microstep: 249.46 | bwd_inner_microstep: 249.11 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 167.03 | bwd_microstep: 285.69 | bwd_inner_microstep: 285.54 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 164.90 | bwd_microstep: 248.26 | bwd_inner_microstep: 247.96 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 166.10 | bwd_microstep: 238.66 | bwd_inner_microstep: 238.53 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 163.59 | bwd_microstep: 277.35 | bwd_inner_microstep: 266.99 | bwd_allreduce_microstep: 10.28 | step_microstep: 21.35 [Rank 0] time (ms) | fwd: 1321.29 | bwd: 2009.97 | bwd_inner: 1997.75 | bwd_allreduce: 11.13 | step: 22.23 11%|█ | 320/2865 [2:11:23<19:08:02, 27.07s/it] {'loss': 0.9107, 'learning_rate': 3.930430266188348e-05, 'epoch': 0.33} 11%|█ | 320/2865 [2:11:23<19:08:02, 27.07s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 163.01 | bwd_microstep: 232.51 | bwd_inner_microstep: 231.70 | bwd_allreduce_microstep: 0.35 | step_microstep: 0.24 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 170.36 | bwd_microstep: 225.49 | bwd_inner_microstep: 225.20 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 167.49 | bwd_microstep: 236.38 | bwd_inner_microstep: 236.19 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 188.95 | bwd_microstep: 230.47 | bwd_inner_microstep: 230.20 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.51 | bwd_microstep: 226.93 | bwd_inner_microstep: 226.59 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 165.92 | bwd_microstep: 223.69 | bwd_inner_microstep: 223.53 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 238.98 | bwd_microstep: 236.05 | bwd_inner_microstep: 235.66 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.30 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.24 | bwd_microstep: 341.53 | bwd_inner_microstep: 331.09 | bwd_allreduce_microstep: 10.37 | step_microstep: 24.30 [Rank 0] time (ms) | fwd: 1419.32 | bwd: 1953.30 | bwd_inner: 1940.61 | bwd_allreduce: 11.37 | step: 25.33 11%|█ | 321/2865 [2:11:27<14:08:56, 20.02s/it] {'loss': 1.0195, 'learning_rate': 3.9298378911631776e-05, 'epoch': 0.34} 11%|█ | 321/2865 [2:11:27<14:08:56, 20.02s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 945 [Rank 0] time (ms) | fwd_microstep: 174.81 | bwd_microstep: 224.52 | bwd_inner_microstep: 224.24 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 192.87 | bwd_microstep: 229.12 | bwd_inner_microstep: 228.76 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 164.51 | bwd_microstep: 232.97 | bwd_inner_microstep: 232.67 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.76 | bwd_microstep: 228.38 | bwd_inner_microstep: 228.01 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.54 | bwd_microstep: 217.52 | bwd_inner_microstep: 217.36 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.89 | bwd_microstep: 232.57 | bwd_inner_microstep: 232.21 | bwd_allreduce_microstep: 0.20 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 164.94 | bwd_microstep: 227.39 | bwd_inner_microstep: 227.18 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.98 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 149.01 | bwd_microstep: 226.55 | bwd_inner_microstep: 216.22 | bwd_allreduce_microstep: 10.17 | step_microstep: 21.27 [Rank 0] time (ms) | fwd: 1330.21 | bwd: 1819.18 | bwd_inner: 1807.07 | bwd_allreduce: 11.05 | step: 22.08 11%|█ | 322/2865 [2:11:30<10:36:25, 15.02s/it] {'loss': 0.9892, 'learning_rate': 3.929243049852023e-05, 'epoch': 0.34} 11%|█ | 322/2865 [2:11:30<10:36:25, 15.02s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 940 [Rank 0] time (ms) | fwd_microstep: 166.72 | bwd_microstep: 233.95 | bwd_inner_microstep: 233.73 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 179.59 | bwd_microstep: 226.21 | bwd_inner_microstep: 225.81 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.25 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 175.52 | bwd_microstep: 218.15 | bwd_inner_microstep: 218.03 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 163.79 | bwd_microstep: 317.95 | bwd_inner_microstep: 317.77 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 169.42 | bwd_microstep: 240.87 | bwd_inner_microstep: 240.71 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 169.44 | bwd_microstep: 248.27 | bwd_inner_microstep: 248.13 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 167.82 | bwd_microstep: 281.89 | bwd_inner_microstep: 281.62 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.07 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 171.28 | bwd_microstep: 305.54 | bwd_inner_microstep: 295.36 | bwd_allreduce_microstep: 10.07 | step_microstep: 22.06 [Rank 0] time (ms) | fwd: 1363.48 | bwd: 2072.97 | bwd_inner: 2061.48 | bwd_allreduce: 10.71 | step: 22.85 11%|█▏ | 323/2865 [2:11:34<8:11:23, 11.60s/it] {'loss': 0.9966, 'learning_rate': 3.928645743015077e-05, 'epoch': 0.34} 11%|█▏ | 323/2865 [2:11:34<8:11:23, 11.60s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 940 [Rank 0] time (ms) | fwd_microstep: 170.09 | bwd_microstep: 241.63 | bwd_inner_microstep: 241.40 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 163.55 | bwd_microstep: 238.44 | bwd_inner_microstep: 238.30 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 172.71 | bwd_microstep: 239.47 | bwd_inner_microstep: 239.27 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 167.49 | bwd_microstep: 288.78 | bwd_inner_microstep: 288.59 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 168.90 | bwd_microstep: 268.20 | bwd_inner_microstep: 268.03 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 168.86 | bwd_microstep: 242.90 | bwd_inner_microstep: 242.77 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 168.46 | bwd_microstep: 264.51 | bwd_inner_microstep: 264.28 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.94 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 165.57 | bwd_microstep: 270.44 | bwd_inner_microstep: 259.09 | bwd_allreduce_microstep: 11.24 | step_microstep: 22.51 [Rank 0] time (ms) | fwd: 1345.51 | bwd: 2054.48 | bwd_inner: 2042.00 | bwd_allreduce: 11.78 | step: 23.19 11%|█▏ | 324/2865 [2:11:37<6:29:18, 9.19s/it] {'loss': 1.0334, 'learning_rate': 3.9280459714156835e-05, 'epoch': 0.34} 11%|█▏ | 324/2865 [2:11:37<6:29:18, 9.19s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 169.91 | bwd_microstep: 234.71 | bwd_inner_microstep: 234.55 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 167.11 | bwd_microstep: 235.07 | bwd_inner_microstep: 234.80 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 163.44 | bwd_microstep: 296.61 | bwd_inner_microstep: 296.49 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 170.54 | bwd_microstep: 243.16 | bwd_inner_microstep: 242.88 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 168.02 | bwd_microstep: 244.21 | bwd_inner_microstep: 244.11 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 168.85 | bwd_microstep: 236.50 | bwd_inner_microstep: 236.31 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 166.77 | bwd_microstep: 273.88 | bwd_inner_microstep: 273.67 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.94 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 167.92 | bwd_microstep: 302.78 | bwd_inner_microstep: 292.43 | bwd_allreduce_microstep: 10.26 | step_microstep: 21.43 [Rank 0] time (ms) | fwd: 1342.46 | bwd: 2067.03 | bwd_inner: 2055.47 | bwd_allreduce: 10.86 | step: 22.08 11%|█▏ | 325/2865 [2:11:41<5:18:06, 7.51s/it] {'loss': 0.9775, 'learning_rate': 3.927443735820336e-05, 'epoch': 0.34} 11%|█▏ | 325/2865 [2:11:41<5:18:06, 7.51s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.14 | bwd_microstep: 266.78 | bwd_inner_microstep: 266.62 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 158.77 | bwd_microstep: 250.95 | bwd_inner_microstep: 250.76 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 202.37 | bwd_microstep: 234.79 | bwd_inner_microstep: 234.71 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.49 | bwd_microstep: 317.08 | bwd_inner_microstep: 316.91 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.01 | bwd_microstep: 256.42 | bwd_inner_microstep: 256.30 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 156.87 | bwd_microstep: 237.95 | bwd_inner_microstep: 237.84 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 157.16 | bwd_microstep: 296.25 | bwd_inner_microstep: 296.10 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 146.86 | bwd_microstep: 261.35 | bwd_inner_microstep: 252.95 | bwd_allreduce_microstep: 8.32 | step_microstep: 21.49 [Rank 0] time (ms) | fwd: 1303.59 | bwd: 2121.64 | bwd_inner: 2112.36 | bwd_allreduce: 8.77 | step: 22.03 11%|█▏ | 326/2865 [2:11:44<4:28:01, 6.33s/it] {'loss': 0.9655, 'learning_rate': 3.926839036998676e-05, 'epoch': 0.34} 11%|█▏ | 326/2865 [2:11:44<4:28:01, 6.33s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 165.45 | bwd_microstep: 233.73 | bwd_inner_microstep: 233.43 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 160.14 | bwd_microstep: 225.63 | bwd_inner_microstep: 225.51 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 161.08 | bwd_microstep: 231.50 | bwd_inner_microstep: 231.36 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 171.82 | bwd_microstep: 237.70 | bwd_inner_microstep: 237.58 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 180.48 | bwd_microstep: 264.53 | bwd_inner_microstep: 264.32 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 166.25 | bwd_microstep: 249.47 | bwd_inner_microstep: 249.33 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 168.68 | bwd_microstep: 234.30 | bwd_inner_microstep: 234.22 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.01 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 165.57 | bwd_microstep: 269.13 | bwd_inner_microstep: 259.33 | bwd_allreduce_microstep: 9.70 | step_microstep: 23.56 [Rank 0] time (ms) | fwd: 1339.34 | bwd: 1946.07 | bwd_inner: 1935.33 | bwd_allreduce: 10.16 | step: 24.07 11%|█▏ | 327/2865 [2:11:48<3:51:27, 5.47s/it] {'loss': 0.9824, 'learning_rate': 3.9262318757234954e-05, 'epoch': 0.34} 11%|█▏ | 327/2865 [2:11:48<3:51:27, 5.47s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 167.91 | bwd_microstep: 238.37 | bwd_inner_microstep: 238.27 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 167.29 | bwd_microstep: 247.96 | bwd_inner_microstep: 247.78 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 175.06 | bwd_microstep: 238.79 | bwd_inner_microstep: 238.56 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 168.39 | bwd_microstep: 227.95 | bwd_inner_microstep: 227.57 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 168.35 | bwd_microstep: 241.66 | bwd_inner_microstep: 241.54 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 171.92 | bwd_microstep: 242.40 | bwd_inner_microstep: 242.16 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 171.84 | bwd_microstep: 244.74 | bwd_inner_microstep: 244.62 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 165.19 | bwd_microstep: 278.69 | bwd_inner_microstep: 268.74 | bwd_allreduce_microstep: 9.86 | step_microstep: 22.31 [Rank 0] time (ms) | fwd: 1355.82 | bwd: 1960.68 | bwd_inner: 1949.50 | bwd_allreduce: 10.47 | step: 23.02 11%|█▏ | 328/2865 [2:11:51<3:26:11, 4.88s/it] {'loss': 0.8625, 'learning_rate': 3.925622252770731e-05, 'epoch': 0.34} 11%|█▏ | 328/2865 [2:11:51<3:26:11, 4.88s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 176.14 | bwd_microstep: 234.00 | bwd_inner_microstep: 233.71 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 186.57 | bwd_microstep: 238.01 | bwd_inner_microstep: 237.78 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 169.68 | bwd_microstep: 233.01 | bwd_inner_microstep: 232.80 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 177.52 | bwd_microstep: 244.15 | bwd_inner_microstep: 243.90 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 166.13 | bwd_microstep: 246.22 | bwd_inner_microstep: 246.04 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 168.02 | bwd_microstep: 237.08 | bwd_inner_microstep: 236.87 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 151.07 | bwd_microstep: 215.34 | bwd_inner_microstep: 215.22 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.97 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 164.72 | bwd_microstep: 272.18 | bwd_inner_microstep: 261.55 | bwd_allreduce_microstep: 10.54 | step_microstep: 21.68 [Rank 0] time (ms) | fwd: 1359.70 | bwd: 1920.11 | bwd_inner: 1908.18 | bwd_allreduce: 11.18 | step: 22.43 11%|█▏ | 329/2865 [2:11:55<3:08:16, 4.45s/it] {'loss': 0.8367, 'learning_rate': 3.9250101689194654e-05, 'epoch': 0.34} 11%|█▏ | 329/2865 [2:11:55<3:08:16, 4.45s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 169.16 | bwd_microstep: 232.80 | bwd_inner_microstep: 232.61 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 168.34 | bwd_microstep: 259.31 | bwd_inner_microstep: 258.97 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 167.60 | bwd_microstep: 237.05 | bwd_inner_microstep: 236.89 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 182.18 | bwd_microstep: 243.50 | bwd_inner_microstep: 243.31 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 165.71 | bwd_microstep: 240.24 | bwd_inner_microstep: 240.12 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 167.51 | bwd_microstep: 236.28 | bwd_inner_microstep: 235.96 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 168.68 | bwd_microstep: 250.19 | bwd_inner_microstep: 250.01 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.86 [Rank 0] time (ms) | fwd_microstep: 164.39 | bwd_microstep: 333.02 | bwd_inner_microstep: 322.78 | bwd_allreduce_microstep: 10.15 | step_microstep: 21.23 [Rank 0] time (ms) | fwd: 1353.43 | bwd: 2032.49 | bwd_inner: 2020.93 | bwd_allreduce: 10.81 | step: 21.91 12%|█▏ | 330/2865 [2:11:58<2:56:55, 4.19s/it] {'loss': 0.7815, 'learning_rate': 3.924395624951928e-05, 'epoch': 0.35} 12%|█▏ | 330/2865 [2:11:58<2:56:55, 4.19s/it][INFO|trainer.py:3242] 2026-05-14 22:32:41,578 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 22:32:41,578 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 22:32:41,578 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-330 [INFO|configuration_utils.py:473] 2026-05-14 22:37:49,936 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-330/config.json [INFO|configuration_utils.py:594] 2026-05-14 22:37:49,944 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-330/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 22:37:55,143 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-330/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 22:37:55,154 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-330/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 22:37:55,164 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-330/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 22:37:55,167 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-330/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 22:38:03,017 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-255] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 305095.91 | bwd_microstep: 244.39 | bwd_inner_microstep: 244.22 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 170.70 | bwd_microstep: 239.36 | bwd_inner_microstep: 239.23 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 180.41 | bwd_microstep: 247.31 | bwd_inner_microstep: 247.18 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 188.78 | bwd_microstep: 241.27 | bwd_inner_microstep: 241.11 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 167.46 | bwd_microstep: 262.46 | bwd_inner_microstep: 262.34 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 166.00 | bwd_microstep: 258.46 | bwd_inner_microstep: 258.27 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 171.00 | bwd_microstep: 264.65 | bwd_inner_microstep: 264.50 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.67 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 152.52 | bwd_microstep: 260.64 | bwd_inner_microstep: 249.82 | bwd_allreduce_microstep: 10.70 | step_microstep: 28.04 [Rank 0] time (ms) | fwd: 306280.18 | bwd: 2018.63 | bwd_inner: 2006.86 | bwd_allreduce: 11.21 | step: 28.62 12%|█▏ | 331/2865 [2:17:24<70:43:55, 100.49s/it] {'loss': 0.9959, 'learning_rate': 3.9237786216534914e-05, 'epoch': 0.35} 12%|█▏ | 331/2865 [2:17:24<70:43:55, 100.49s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 171.52 | bwd_microstep: 252.33 | bwd_inner_microstep: 252.03 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 170.76 | bwd_microstep: 245.14 | bwd_inner_microstep: 244.98 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 166.42 | bwd_microstep: 243.85 | bwd_inner_microstep: 243.72 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 166.47 | bwd_microstep: 389.30 | bwd_inner_microstep: 389.13 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 188.03 | bwd_microstep: 303.21 | bwd_inner_microstep: 303.05 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 174.62 | bwd_microstep: 247.80 | bwd_inner_microstep: 247.66 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 169.16 | bwd_microstep: 251.00 | bwd_inner_microstep: 250.83 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.02 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 174.95 | bwd_microstep: 292.64 | bwd_inner_microstep: 278.65 | bwd_allreduce_microstep: 13.86 | step_microstep: 23.36 [Rank 0] time (ms) | fwd: 1381.78 | bwd: 2225.38 | bwd_inner: 2210.30 | bwd_allreduce: 14.41 | step: 24.00 12%|█▏ | 332/2865 [2:17:27<50:17:39, 71.48s/it] {'loss': 1.1475, 'learning_rate': 3.923159159812669e-05, 'epoch': 0.35} 12%|█▏ | 332/2865 [2:17:27<50:17:39, 71.48s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 165.33 | bwd_microstep: 253.29 | bwd_inner_microstep: 253.14 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 167.82 | bwd_microstep: 240.92 | bwd_inner_microstep: 240.80 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 173.45 | bwd_microstep: 253.71 | bwd_inner_microstep: 253.55 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 169.74 | bwd_microstep: 247.02 | bwd_inner_microstep: 246.78 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 168.90 | bwd_microstep: 243.08 | bwd_inner_microstep: 242.95 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 169.98 | bwd_microstep: 275.18 | bwd_inner_microstep: 275.05 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 169.25 | bwd_microstep: 242.91 | bwd_inner_microstep: 242.73 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.97 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 179.28 | bwd_microstep: 298.84 | bwd_inner_microstep: 284.21 | bwd_allreduce_microstep: 14.49 | step_microstep: 23.51 [Rank 0] time (ms) | fwd: 1363.62 | bwd: 2055.05 | bwd_inner: 2039.43 | bwd_allreduce: 14.99 | step: 24.10 12%|█▏ | 333/2865 [2:17:31<35:57:08, 51.12s/it] {'loss': 0.9701, 'learning_rate': 3.92253724022112e-05, 'epoch': 0.35} 12%|█▏ | 333/2865 [2:17:31<35:57:08, 51.12s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 180.87 | bwd_microstep: 289.88 | bwd_inner_microstep: 289.74 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 164.34 | bwd_microstep: 325.24 | bwd_inner_microstep: 324.93 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 162.83 | bwd_microstep: 250.49 | bwd_inner_microstep: 250.33 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 168.98 | bwd_microstep: 245.29 | bwd_inner_microstep: 244.95 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 169.50 | bwd_microstep: 240.71 | bwd_inner_microstep: 240.59 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 162.98 | bwd_microstep: 238.91 | bwd_inner_microstep: 238.75 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 170.26 | bwd_microstep: 248.37 | bwd_inner_microstep: 248.10 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.30 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 204.06 | bwd_microstep: 299.70 | bwd_inner_microstep: 284.83 | bwd_allreduce_microstep: 14.76 | step_microstep: 24.88 [Rank 0] time (ms) | fwd: 1383.72 | bwd: 2138.73 | bwd_inner: 2122.48 | bwd_allreduce: 15.43 | step: 25.61 12%|█▏ | 334/2865 [2:17:35<25:56:25, 36.90s/it] {'loss': 0.9098, 'learning_rate': 3.9219128636736413e-05, 'epoch': 0.35} 12%|█▏ | 334/2865 [2:17:35<25:56:25, 36.90s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 185.41 | bwd_microstep: 261.91 | bwd_inner_microstep: 261.70 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 184.34 | bwd_microstep: 269.77 | bwd_inner_microstep: 269.55 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 178.40 | bwd_microstep: 249.01 | bwd_inner_microstep: 248.71 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 171.09 | bwd_microstep: 247.39 | bwd_inner_microstep: 247.25 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 186.52 | bwd_microstep: 255.27 | bwd_inner_microstep: 255.10 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 173.96 | bwd_microstep: 266.40 | bwd_inner_microstep: 266.18 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 171.87 | bwd_microstep: 254.14 | bwd_inner_microstep: 253.98 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.96 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 170.82 | bwd_microstep: 302.29 | bwd_inner_microstep: 291.96 | bwd_allreduce_microstep: 10.26 | step_microstep: 21.58 [Rank 0] time (ms) | fwd: 1422.28 | bwd: 2106.30 | bwd_inner: 2094.70 | bwd_allreduce: 10.90 | step: 22.27 12%|█▏ | 335/2865 [2:17:38<18:56:10, 26.94s/it] {'loss': 0.8714, 'learning_rate': 3.9212860309681706e-05, 'epoch': 0.35} 12%|█▏ | 335/2865 [2:17:38<18:56:10, 26.94s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 171.73 | bwd_microstep: 249.57 | bwd_inner_microstep: 249.42 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 166.39 | bwd_microstep: 248.88 | bwd_inner_microstep: 248.74 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 183.08 | bwd_microstep: 261.16 | bwd_inner_microstep: 261.02 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 169.51 | bwd_microstep: 245.83 | bwd_inner_microstep: 245.65 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 169.63 | bwd_microstep: 245.66 | bwd_inner_microstep: 245.49 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 173.35 | bwd_microstep: 237.56 | bwd_inner_microstep: 237.37 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 169.15 | bwd_microstep: 243.08 | bwd_inner_microstep: 242.92 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.98 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 164.52 | bwd_microstep: 282.46 | bwd_inner_microstep: 267.38 | bwd_allreduce_microstep: 14.96 | step_microstep: 23.67 [Rank 0] time (ms) | fwd: 1367.24 | bwd: 2014.32 | bwd_inner: 1998.22 | bwd_allreduce: 15.46 | step: 24.31 12%|█▏ | 336/2865 [2:17:42<13:59:57, 19.93s/it] {'loss': 0.9181, 'learning_rate': 3.9206567429057866e-05, 'epoch': 0.35} 12%|█▏ | 336/2865 [2:17:42<13:59:57, 19.93s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 164.09 | bwd_microstep: 250.89 | bwd_inner_microstep: 250.76 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 179.30 | bwd_microstep: 234.22 | bwd_inner_microstep: 234.01 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 167.17 | bwd_microstep: 241.00 | bwd_inner_microstep: 240.85 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 163.05 | bwd_microstep: 242.60 | bwd_inner_microstep: 242.47 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 166.15 | bwd_microstep: 240.61 | bwd_inner_microstep: 240.45 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 168.99 | bwd_microstep: 244.54 | bwd_inner_microstep: 244.32 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 172.15 | bwd_microstep: 237.98 | bwd_inner_microstep: 237.85 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.30 | optimizer_step: 4.86 [Rank 0] time (ms) | fwd_microstep: 162.83 | bwd_microstep: 300.17 | bwd_inner_microstep: 286.22 | bwd_allreduce_microstep: 13.85 | step_microstep: 22.45 [Rank 0] time (ms) | fwd: 1343.60 | bwd: 1992.12 | bwd_inner: 1977.15 | bwd_allreduce: 14.35 | step: 23.13 12%|█▏ | 337/2865 [2:17:45<10:32:13, 15.01s/it] {'loss': 0.8804, 'learning_rate': 3.9200250002907025e-05, 'epoch': 0.35} 12%|█▏ | 337/2865 [2:17:45<10:32:13, 15.01s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 187.22 | bwd_microstep: 235.04 | bwd_inner_microstep: 234.85 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 168.84 | bwd_microstep: 243.81 | bwd_inner_microstep: 243.69 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 163.41 | bwd_microstep: 241.68 | bwd_inner_microstep: 241.55 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 168.10 | bwd_microstep: 242.47 | bwd_inner_microstep: 242.34 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 168.59 | bwd_microstep: 240.89 | bwd_inner_microstep: 240.60 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 189.55 | bwd_microstep: 238.63 | bwd_inner_microstep: 238.28 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 169.41 | bwd_microstep: 239.88 | bwd_inner_microstep: 239.58 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 154.46 | bwd_microstep: 253.42 | bwd_inner_microstep: 238.58 | bwd_allreduce_microstep: 14.71 | step_microstep: 21.19 [Rank 0] time (ms) | fwd: 1369.46 | bwd: 1935.95 | bwd_inner: 1919.76 | bwd_allreduce: 15.38 | step: 21.78 12%|█▏ | 338/2865 [2:17:49<8:06:23, 11.55s/it] {'loss': 0.9004, 'learning_rate': 3.919390803930271e-05, 'epoch': 0.35} 12%|█▏ | 338/2865 [2:17:49<8:06:23, 11.55s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 165.78 | bwd_microstep: 238.78 | bwd_inner_microstep: 238.59 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 168.17 | bwd_microstep: 238.49 | bwd_inner_microstep: 238.33 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 171.48 | bwd_microstep: 238.47 | bwd_inner_microstep: 238.34 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 166.96 | bwd_microstep: 241.42 | bwd_inner_microstep: 241.29 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 164.81 | bwd_microstep: 311.33 | bwd_inner_microstep: 311.00 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 162.17 | bwd_microstep: 234.61 | bwd_inner_microstep: 234.48 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 166.49 | bwd_microstep: 238.42 | bwd_inner_microstep: 238.24 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 168.91 | bwd_microstep: 270.81 | bwd_inner_microstep: 261.09 | bwd_allreduce_microstep: 9.64 | step_microstep: 21.09 [Rank 0] time (ms) | fwd: 1334.64 | bwd: 2012.40 | bwd_inner: 2001.63 | bwd_allreduce: 10.16 | step: 21.74 12%|█▏ | 339/2865 [2:17:53<6:24:53, 9.14s/it] {'loss': 0.7742, 'learning_rate': 3.918754154634979e-05, 'epoch': 0.35} 12%|█▏ | 339/2865 [2:17:53<6:24:53, 9.14s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 168.44 | bwd_microstep: 234.43 | bwd_inner_microstep: 234.31 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 170.08 | bwd_microstep: 230.07 | bwd_inner_microstep: 229.86 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 167.52 | bwd_microstep: 234.59 | bwd_inner_microstep: 234.46 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 164.23 | bwd_microstep: 231.19 | bwd_inner_microstep: 231.12 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 164.10 | bwd_microstep: 237.64 | bwd_inner_microstep: 237.56 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 176.53 | bwd_microstep: 234.57 | bwd_inner_microstep: 234.50 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 167.89 | bwd_microstep: 241.65 | bwd_inner_microstep: 241.40 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.44 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 170.35 | bwd_microstep: 288.48 | bwd_inner_microstep: 274.54 | bwd_allreduce_microstep: 13.82 | step_microstep: 26.51 [Rank 0] time (ms) | fwd: 1349.01 | bwd: 1932.72 | bwd_inner: 1917.91 | bwd_allreduce: 14.29 | step: 27.10 12%|█▏ | 340/2865 [2:17:56<5:12:57, 7.44s/it] {'loss': 1.0795, 'learning_rate': 3.91811505321845e-05, 'epoch': 0.36} 12%|█▏ | 340/2865 [2:17:56<5:12:57, 7.44s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 165.15 | bwd_microstep: 250.05 | bwd_inner_microstep: 249.95 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 168.94 | bwd_microstep: 232.35 | bwd_inner_microstep: 232.23 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 168.99 | bwd_microstep: 265.58 | bwd_inner_microstep: 265.48 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 177.28 | bwd_microstep: 247.57 | bwd_inner_microstep: 247.44 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 166.48 | bwd_microstep: 228.42 | bwd_inner_microstep: 228.32 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 164.75 | bwd_microstep: 237.89 | bwd_inner_microstep: 237.69 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 150.78 | bwd_microstep: 205.64 | bwd_inner_microstep: 205.47 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 168.32 | bwd_microstep: 279.23 | bwd_inner_microstep: 269.40 | bwd_allreduce_microstep: 9.73 | step_microstep: 21.18 [Rank 0] time (ms) | fwd: 1330.59 | bwd: 1946.83 | bwd_inner: 1936.09 | bwd_allreduce: 10.22 | step: 21.76 12%|█▏ | 341/2865 [2:17:59<4:22:22, 6.24s/it] {'loss': 0.7658, 'learning_rate': 3.917473500497438e-05, 'epoch': 0.36} 12%|█▏ | 341/2865 [2:17:59<4:22:22, 6.24s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 179.26 | bwd_microstep: 241.93 | bwd_inner_microstep: 241.64 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 168.09 | bwd_microstep: 256.90 | bwd_inner_microstep: 256.69 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 166.00 | bwd_microstep: 238.43 | bwd_inner_microstep: 238.29 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 167.90 | bwd_microstep: 234.69 | bwd_inner_microstep: 234.42 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 166.72 | bwd_microstep: 232.90 | bwd_inner_microstep: 232.69 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 165.16 | bwd_microstep: 234.48 | bwd_inner_microstep: 234.39 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 149.91 | bwd_microstep: 198.86 | bwd_inner_microstep: 198.59 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 155.26 | bwd_microstep: 241.11 | bwd_inner_microstep: 226.51 | bwd_allreduce_microstep: 14.51 | step_microstep: 21.33 [Rank 0] time (ms) | fwd: 1318.19 | bwd: 1879.44 | bwd_inner: 1863.49 | bwd_allreduce: 15.18 | step: 21.96 12%|█▏ | 342/2865 [2:18:03<3:46:14, 5.38s/it] {'loss': 0.9203, 'learning_rate': 3.916829497291834e-05, 'epoch': 0.36} 12%|█▏ | 342/2865 [2:18:03<3:46:14, 5.38s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 163.12 | bwd_microstep: 233.11 | bwd_inner_microstep: 232.77 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 168.53 | bwd_microstep: 255.54 | bwd_inner_microstep: 255.34 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 166.43 | bwd_microstep: 201.95 | bwd_inner_microstep: 201.64 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 169.66 | bwd_microstep: 200.64 | bwd_inner_microstep: 200.56 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 168.70 | bwd_microstep: 204.33 | bwd_inner_microstep: 203.99 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 167.58 | bwd_microstep: 242.68 | bwd_inner_microstep: 242.54 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 168.22 | bwd_microstep: 235.27 | bwd_inner_microstep: 235.14 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.36 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.78 | bwd_microstep: 282.10 | bwd_inner_microstep: 269.47 | bwd_allreduce_microstep: 12.43 | step_microstep: 25.52 [Rank 0] time (ms) | fwd: 1333.90 | bwd: 1855.77 | bwd_inner: 1841.78 | bwd_allreduce: 13.10 | step: 26.35 12%|█▏ | 343/2865 [2:18:06<3:20:59, 4.78s/it] {'loss': 0.8532, 'learning_rate': 3.916183044424657e-05, 'epoch': 0.36} 12%|█▏ | 343/2865 [2:18:06<3:20:59, 4.78s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 152.70 | bwd_microstep: 247.45 | bwd_inner_microstep: 247.25 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 164.19 | bwd_microstep: 234.60 | bwd_inner_microstep: 234.46 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 166.44 | bwd_microstep: 236.21 | bwd_inner_microstep: 236.08 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 168.69 | bwd_microstep: 237.18 | bwd_inner_microstep: 236.95 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 167.03 | bwd_microstep: 245.06 | bwd_inner_microstep: 244.93 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 164.78 | bwd_microstep: 267.39 | bwd_inner_microstep: 267.25 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 154.45 | bwd_microstep: 209.43 | bwd_inner_microstep: 209.32 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 172.17 | bwd_microstep: 276.03 | bwd_inner_microstep: 266.24 | bwd_allreduce_microstep: 9.71 | step_microstep: 20.98 [Rank 0] time (ms) | fwd: 1310.35 | bwd: 1953.45 | bwd_inner: 1942.68 | bwd_allreduce: 10.19 | step: 21.57 12%|█▏ | 344/2865 [2:18:10<3:03:56, 4.38s/it] {'loss': 1.0076, 'learning_rate': 3.9155341427220596e-05, 'epoch': 0.36} 12%|█▏ | 344/2865 [2:18:10<3:03:56, 4.38s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 172.49 | bwd_microstep: 238.65 | bwd_inner_microstep: 238.46 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 168.71 | bwd_microstep: 245.47 | bwd_inner_microstep: 245.33 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 163.84 | bwd_microstep: 232.86 | bwd_inner_microstep: 232.76 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 162.46 | bwd_microstep: 236.54 | bwd_inner_microstep: 236.39 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 167.92 | bwd_microstep: 244.33 | bwd_inner_microstep: 244.19 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 169.84 | bwd_microstep: 249.07 | bwd_inner_microstep: 248.92 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 151.86 | bwd_microstep: 206.27 | bwd_inner_microstep: 205.93 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.95 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 166.92 | bwd_microstep: 282.78 | bwd_inner_microstep: 272.38 | bwd_allreduce_microstep: 10.31 | step_microstep: 22.04 [Rank 0] time (ms) | fwd: 1323.95 | bwd: 1936.07 | bwd_inner: 1924.59 | bwd_allreduce: 10.84 | step: 22.66 12%|█▏ | 345/2865 [2:18:13<2:52:08, 4.10s/it] {'loss': 0.9221, 'learning_rate': 3.91488279301332e-05, 'epoch': 0.36} 12%|█▏ | 345/2865 [2:18:13<2:52:08, 4.10s/it][INFO|trainer.py:3242] 2026-05-14 22:38:56,234 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 22:38:56,235 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 22:38:56,235 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-345 [INFO|configuration_utils.py:473] 2026-05-14 22:44:03,977 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-345/config.json [INFO|configuration_utils.py:594] 2026-05-14 22:44:03,984 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-345/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 22:44:08,269 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-345/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 22:44:08,281 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-345/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 22:44:08,288 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-345/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 22:44:08,290 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-345/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 22:44:14,749 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-270] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 304397.11 | bwd_microstep: 237.44 | bwd_inner_microstep: 237.25 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 166.46 | bwd_microstep: 248.78 | bwd_inner_microstep: 248.66 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.68 | bwd_microstep: 235.96 | bwd_inner_microstep: 235.88 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 166.72 | bwd_microstep: 238.19 | bwd_inner_microstep: 238.05 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 162.56 | bwd_microstep: 237.22 | bwd_inner_microstep: 237.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 166.43 | bwd_microstep: 237.15 | bwd_inner_microstep: 237.05 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 180.57 | bwd_microstep: 240.39 | bwd_inner_microstep: 240.20 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.98 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 177.87 | bwd_microstep: 275.30 | bwd_inner_microstep: 265.30 | bwd_allreduce_microstep: 9.92 | step_microstep: 21.51 [Rank 0] time (ms) | fwd: 305566.68 | bwd: 1950.52 | bwd_inner: 1939.61 | bwd_allreduce: 10.43 | step: 22.13 12%|█▏ | 346/2865 [2:23:35<69:39:22, 99.55s/it] {'loss': 0.8362, 'learning_rate': 3.9142289961308496e-05, 'epoch': 0.36} 12%|█▏ | 346/2865 [2:23:35<69:39:22, 99.55s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 174.11 | bwd_microstep: 229.56 | bwd_inner_microstep: 229.37 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 177.90 | bwd_microstep: 236.87 | bwd_inner_microstep: 236.68 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.78 | bwd_microstep: 249.39 | bwd_inner_microstep: 249.05 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 167.43 | bwd_microstep: 233.72 | bwd_inner_microstep: 233.59 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 170.04 | bwd_microstep: 244.47 | bwd_inner_microstep: 244.34 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 165.63 | bwd_microstep: 234.11 | bwd_inner_microstep: 234.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 147.56 | bwd_microstep: 206.73 | bwd_inner_microstep: 206.59 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 177.56 | bwd_microstep: 270.21 | bwd_inner_microstep: 260.37 | bwd_allreduce_microstep: 9.77 | step_microstep: 21.33 [Rank 0] time (ms) | fwd: 1340.89 | bwd: 1905.18 | bwd_inner: 1894.24 | bwd_allreduce: 10.31 | step: 22.03 12%|█▏ | 347/2865 [2:23:39<49:27:42, 70.72s/it] {'loss': 0.8222, 'learning_rate': 3.913572752910185e-05, 'epoch': 0.36} 12%|█▏ | 347/2865 [2:23:39<49:27:42, 70.72s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 164.41 | bwd_microstep: 244.90 | bwd_inner_microstep: 244.77 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.93 | bwd_microstep: 242.08 | bwd_inner_microstep: 241.95 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 168.42 | bwd_microstep: 236.61 | bwd_inner_microstep: 236.25 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 170.23 | bwd_microstep: 262.59 | bwd_inner_microstep: 262.45 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 163.40 | bwd_microstep: 257.15 | bwd_inner_microstep: 257.02 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 168.18 | bwd_microstep: 254.82 | bwd_inner_microstep: 254.63 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 193.12 | bwd_microstep: 250.84 | bwd_inner_microstep: 250.60 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.93 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 169.15 | bwd_microstep: 278.72 | bwd_inner_microstep: 268.47 | bwd_allreduce_microstep: 10.14 | step_microstep: 21.13 [Rank 0] time (ms) | fwd: 1360.74 | bwd: 2027.84 | bwd_inner: 2016.44 | bwd_allreduce: 10.74 | step: 21.84 12%|█▏ | 348/2865 [2:23:42<35:21:28, 50.57s/it] {'loss': 0.9317, 'learning_rate': 3.912914064189987e-05, 'epoch': 0.36} 12%|█▏ | 348/2865 [2:23:42<35:21:28, 50.57s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 169.86 | bwd_microstep: 262.18 | bwd_inner_microstep: 262.05 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 167.53 | bwd_microstep: 216.06 | bwd_inner_microstep: 215.88 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 174.28 | bwd_microstep: 271.87 | bwd_inner_microstep: 271.71 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 173.33 | bwd_microstep: 256.52 | bwd_inner_microstep: 256.37 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 170.43 | bwd_microstep: 215.11 | bwd_inner_microstep: 215.01 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 171.11 | bwd_microstep: 214.13 | bwd_inner_microstep: 213.94 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 175.11 | bwd_microstep: 229.64 | bwd_inner_microstep: 229.57 | bwd_allreduce_microstep: 0.03 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 169.85 | bwd_microstep: 260.73 | bwd_inner_microstep: 251.01 | bwd_allreduce_microstep: 9.63 | step_microstep: 20.84 [Rank 0] time (ms) | fwd: 1371.34 | bwd: 1926.35 | bwd_inner: 1915.71 | bwd_allreduce: 10.11 | step: 21.47 12%|█▏ | 349/2865 [2:23:46<25:28:16, 36.45s/it] {'loss': 0.9229, 'learning_rate': 3.912252930812045e-05, 'epoch': 0.37} 12%|█▏ | 349/2865 [2:23:46<25:28:16, 36.45s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 169.91 | bwd_microstep: 245.43 | bwd_inner_microstep: 245.21 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 167.94 | bwd_microstep: 243.87 | bwd_inner_microstep: 243.71 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 942 [Rank 0] time (ms) | fwd_microstep: 166.10 | bwd_microstep: 243.62 | bwd_inner_microstep: 243.48 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 942 [Rank 0] time (ms) | fwd_microstep: 164.05 | bwd_microstep: 239.75 | bwd_inner_microstep: 239.55 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 172.05 | bwd_microstep: 233.79 | bwd_inner_microstep: 233.42 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.26 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 172.13 | bwd_microstep: 246.38 | bwd_inner_microstep: 246.24 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 166.68 | bwd_microstep: 257.79 | bwd_inner_microstep: 257.56 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.44 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 175.09 | bwd_microstep: 285.76 | bwd_inner_microstep: 273.82 | bwd_allreduce_microstep: 11.83 | step_microstep: 28.53 [Rank 0] time (ms) | fwd: 1353.81 | bwd: 1996.55 | bwd_inner: 1983.25 | bwd_allreduce: 12.48 | step: 29.45 12%|█▏ | 350/2865 [2:23:49<18:34:11, 26.58s/it] {'loss': 1.0065, 'learning_rate': 3.911589353621271e-05, 'epoch': 0.37} 12%|█▏ | 350/2865 [2:23:49<18:34:11, 26.58s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 175.39 | bwd_microstep: 269.11 | bwd_inner_microstep: 268.93 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 176.20 | bwd_microstep: 276.66 | bwd_inner_microstep: 276.47 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 190.16 | bwd_microstep: 321.76 | bwd_inner_microstep: 321.57 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 166.63 | bwd_microstep: 263.40 | bwd_inner_microstep: 263.24 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 188.74 | bwd_microstep: 266.48 | bwd_inner_microstep: 266.18 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.74 | bwd_microstep: 266.12 | bwd_inner_microstep: 265.70 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 156.39 | bwd_microstep: 207.33 | bwd_inner_microstep: 207.17 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.98 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 165.75 | bwd_microstep: 296.65 | bwd_inner_microstep: 285.76 | bwd_allreduce_microstep: 10.72 | step_microstep: 21.57 [Rank 0] time (ms) | fwd: 1382.86 | bwd: 2167.61 | bwd_inner: 2155.42 | bwd_allreduce: 11.36 | step: 22.36 12%|█▏ | 351/2865 [2:23:53<13:46:46, 19.73s/it] {'loss': 0.7981, 'learning_rate': 3.910923333465702e-05, 'epoch': 0.37} 12%|█▏ | 351/2865 [2:23:53<13:46:46, 19.73s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 953 [Rank 0] time (ms) | fwd_microstep: 179.37 | bwd_microstep: 296.89 | bwd_inner_microstep: 296.72 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 177.50 | bwd_microstep: 283.75 | bwd_inner_microstep: 283.44 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 200.55 | bwd_microstep: 245.77 | bwd_inner_microstep: 245.58 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 165.78 | bwd_microstep: 279.36 | bwd_inner_microstep: 279.19 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 180.15 | bwd_microstep: 237.15 | bwd_inner_microstep: 237.03 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 164.20 | bwd_microstep: 237.54 | bwd_inner_microstep: 237.35 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 183.69 | bwd_microstep: 235.67 | bwd_inner_microstep: 235.29 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.38 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 179.19 | bwd_microstep: 313.35 | bwd_inner_microstep: 302.52 | bwd_allreduce_microstep: 10.67 | step_microstep: 22.56 [Rank 0] time (ms) | fwd: 1430.32 | bwd: 2129.59 | bwd_inner: 2117.47 | bwd_allreduce: 11.29 | step: 23.38 12%|█▏ | 352/2865 [2:23:57<10:25:35, 14.94s/it] {'loss': 1.0924, 'learning_rate': 3.910254871196494e-05, 'epoch': 0.37} 12%|█▏ | 352/2865 [2:23:57<10:25:35, 14.94s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 182.36 | bwd_microstep: 274.56 | bwd_inner_microstep: 274.38 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 182.70 | bwd_microstep: 324.68 | bwd_inner_microstep: 324.27 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 181.29 | bwd_microstep: 307.48 | bwd_inner_microstep: 307.27 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 165.27 | bwd_microstep: 245.46 | bwd_inner_microstep: 245.07 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 170.73 | bwd_microstep: 227.17 | bwd_inner_microstep: 227.00 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 164.67 | bwd_microstep: 236.46 | bwd_inner_microstep: 236.13 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 165.14 | bwd_microstep: 263.61 | bwd_inner_microstep: 263.23 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 163.65 | bwd_microstep: 287.37 | bwd_inner_microstep: 276.98 | bwd_allreduce_microstep: 10.18 | step_microstep: 21.00 [Rank 0] time (ms) | fwd: 1375.64 | bwd: 2166.98 | bwd_inner: 2154.79 | bwd_allreduce: 11.09 | step: 21.96 12%|█▏ | 353/2865 [2:24:01<8:04:49, 11.58s/it] {'loss': 0.7513, 'learning_rate': 3.909583967667927e-05, 'epoch': 0.37} 12%|█▏ | 353/2865 [2:24:01<8:04:49, 11.58s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 171.60 | bwd_microstep: 232.22 | bwd_inner_microstep: 232.04 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 167.22 | bwd_microstep: 230.84 | bwd_inner_microstep: 230.58 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 180.92 | bwd_microstep: 236.98 | bwd_inner_microstep: 236.51 | bwd_allreduce_microstep: 0.22 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 181.81 | bwd_microstep: 272.31 | bwd_inner_microstep: 272.12 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 170.72 | bwd_microstep: 235.05 | bwd_inner_microstep: 234.90 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 166.10 | bwd_microstep: 269.91 | bwd_inner_microstep: 269.53 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 168.56 | bwd_microstep: 266.54 | bwd_inner_microstep: 266.35 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 164.91 | bwd_microstep: 276.68 | bwd_inner_microstep: 266.48 | bwd_allreduce_microstep: 10.10 | step_microstep: 21.16 [Rank 0] time (ms) | fwd: 1371.71 | bwd: 2020.71 | bwd_inner: 2008.82 | bwd_allreduce: 10.93 | step: 21.99 12%|█▏ | 354/2865 [2:24:04<6:24:19, 9.18s/it] {'loss': 1.1365, 'learning_rate': 3.908910623737398e-05, 'epoch': 0.37} 12%|█▏ | 354/2865 [2:24:04<6:24:19, 9.18s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 167.95 | bwd_microstep: 243.56 | bwd_inner_microstep: 243.34 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 177.06 | bwd_microstep: 253.92 | bwd_inner_microstep: 253.79 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 175.80 | bwd_microstep: 280.75 | bwd_inner_microstep: 280.48 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 162.03 | bwd_microstep: 220.13 | bwd_inner_microstep: 219.98 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 164.07 | bwd_microstep: 243.94 | bwd_inner_microstep: 243.57 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.98 | bwd_microstep: 225.22 | bwd_inner_microstep: 225.04 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 168.68 | bwd_microstep: 245.43 | bwd_inner_microstep: 245.18 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 164.23 | bwd_microstep: 260.73 | bwd_inner_microstep: 249.63 | bwd_allreduce_microstep: 10.98 | step_microstep: 21.30 [Rank 0] time (ms) | fwd: 1340.70 | bwd: 1973.80 | bwd_inner: 1961.32 | bwd_allreduce: 11.63 | step: 22.03 12%|█▏ | 355/2865 [2:24:08<5:12:44, 7.48s/it] {'loss': 0.9826, 'learning_rate': 3.908234840265424e-05, 'epoch': 0.37} 12%|█▏ | 355/2865 [2:24:08<5:12:44, 7.48s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 166.35 | bwd_microstep: 220.87 | bwd_inner_microstep: 220.55 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 162.60 | bwd_microstep: 228.24 | bwd_inner_microstep: 228.13 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 163.65 | bwd_microstep: 222.65 | bwd_inner_microstep: 222.38 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 166.81 | bwd_microstep: 223.65 | bwd_inner_microstep: 223.47 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.04 | bwd_microstep: 219.29 | bwd_inner_microstep: 219.09 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 164.67 | bwd_microstep: 256.27 | bwd_inner_microstep: 255.98 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.47 | bwd_microstep: 223.77 | bwd_inner_microstep: 223.50 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.27 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 161.92 | bwd_microstep: 264.17 | bwd_inner_microstep: 251.22 | bwd_allreduce_microstep: 12.82 | step_microstep: 22.21 [Rank 0] time (ms) | fwd: 1308.39 | bwd: 1859.06 | bwd_inner: 1844.67 | bwd_allreduce: 13.49 | step: 23.07 12%|█▏ | 356/2865 [2:24:11<4:21:01, 6.24s/it] {'loss': 1.0868, 'learning_rate': 3.9075566181156416e-05, 'epoch': 0.37} 12%|█▏ | 356/2865 [2:24:11<4:21:01, 6.24s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 944 [Rank 0] time (ms) | fwd_microstep: 171.96 | bwd_microstep: 229.88 | bwd_inner_microstep: 229.72 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 162.27 | bwd_microstep: 229.55 | bwd_inner_microstep: 229.43 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 164.64 | bwd_microstep: 230.00 | bwd_inner_microstep: 229.82 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 173.09 | bwd_microstep: 230.38 | bwd_inner_microstep: 230.09 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 167.09 | bwd_microstep: 225.77 | bwd_inner_microstep: 225.67 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 166.42 | bwd_microstep: 243.36 | bwd_inner_microstep: 243.04 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 150.82 | bwd_microstep: 241.06 | bwd_inner_microstep: 240.80 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.96 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 195.51 | bwd_microstep: 317.05 | bwd_inner_microstep: 306.05 | bwd_allreduce_microstep: 10.90 | step_microstep: 20.99 [Rank 0] time (ms) | fwd: 1351.66 | bwd: 1947.16 | bwd_inner: 1934.90 | bwd_allreduce: 11.51 | step: 21.71 12%|█▏ | 357/2865 [2:24:15<3:46:25, 5.42s/it] {'loss': 0.9694, 'learning_rate': 3.906875958154801e-05, 'epoch': 0.37} 12%|█▏ | 357/2865 [2:24:15<3:46:25, 5.42s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 164.36 | bwd_microstep: 282.31 | bwd_inner_microstep: 282.12 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 175.57 | bwd_microstep: 281.65 | bwd_inner_microstep: 280.96 | bwd_allreduce_microstep: 0.29 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 178.16 | bwd_microstep: 278.58 | bwd_inner_microstep: 278.39 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 163.19 | bwd_microstep: 247.78 | bwd_inner_microstep: 247.51 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 167.12 | bwd_microstep: 232.07 | bwd_inner_microstep: 231.67 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 168.08 | bwd_microstep: 233.49 | bwd_inner_microstep: 233.12 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 166.17 | bwd_microstep: 303.33 | bwd_inner_microstep: 303.16 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.95 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 177.12 | bwd_microstep: 315.19 | bwd_inner_microstep: 298.51 | bwd_allreduce_microstep: 16.55 | step_microstep: 21.38 [Rank 0] time (ms) | fwd: 1359.64 | bwd: 2174.63 | bwd_inner: 2155.91 | bwd_allreduce: 17.51 | step: 22.32 12%|█▏ | 358/2865 [2:24:18<3:25:16, 4.91s/it] {'loss': 1.0107, 'learning_rate': 3.906192861252769e-05, 'epoch': 0.37} 12%|█▏ | 358/2865 [2:24:18<3:25:16, 4.91s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 168.61 | bwd_microstep: 292.73 | bwd_inner_microstep: 292.51 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 192.44 | bwd_microstep: 289.91 | bwd_inner_microstep: 289.73 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 180.30 | bwd_microstep: 242.24 | bwd_inner_microstep: 242.01 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 168.74 | bwd_microstep: 270.39 | bwd_inner_microstep: 270.21 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 162.77 | bwd_microstep: 224.64 | bwd_inner_microstep: 224.51 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 172.17 | bwd_microstep: 287.09 | bwd_inner_microstep: 286.88 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 229.18 | bwd_microstep: 211.08 | bwd_inner_microstep: 210.94 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.38 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 185.48 | bwd_microstep: 335.24 | bwd_inner_microstep: 321.27 | bwd_allreduce_microstep: 13.83 | step_microstep: 24.58 [Rank 0] time (ms) | fwd: 1459.55 | bwd: 2153.40 | bwd_inner: 2138.35 | bwd_allreduce: 14.38 | step: 25.23 13%|█▎ | 359/2865 [2:24:22<3:11:37, 4.59s/it] {'loss': 0.6974, 'learning_rate': 3.9055073282825274e-05, 'epoch': 0.38} 13%|█▎ | 359/2865 [2:24:22<3:11:37, 4.59s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 192.79 | bwd_microstep: 260.69 | bwd_inner_microstep: 260.47 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 173.58 | bwd_microstep: 246.14 | bwd_inner_microstep: 246.01 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 166.89 | bwd_microstep: 245.52 | bwd_inner_microstep: 244.98 | bwd_allreduce_microstep: 0.30 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 169.40 | bwd_microstep: 232.78 | bwd_inner_microstep: 232.55 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 166.44 | bwd_microstep: 253.70 | bwd_inner_microstep: 253.55 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 166.31 | bwd_microstep: 266.93 | bwd_inner_microstep: 266.63 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 165.29 | bwd_microstep: 242.99 | bwd_inner_microstep: 242.83 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 167.22 | bwd_microstep: 296.01 | bwd_inner_microstep: 286.06 | bwd_allreduce_microstep: 9.88 | step_microstep: 21.39 [Rank 0] time (ms) | fwd: 1367.78 | bwd: 2044.90 | bwd_inner: 2033.38 | bwd_allreduce: 10.73 | step: 22.14 13%|█▎ | 360/2865 [2:24:26<2:59:17, 4.29s/it] {'loss': 0.9738, 'learning_rate': 3.904819360120171e-05, 'epoch': 0.38} 13%|█▎ | 360/2865 [2:24:26<2:59:17, 4.29s/it][INFO|trainer.py:3242] 2026-05-14 22:45:08,915 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 22:45:08,915 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 22:45:08,916 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-360 [INFO|configuration_utils.py:473] 2026-05-14 22:50:16,521 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-360/config.json [INFO|configuration_utils.py:594] 2026-05-14 22:50:16,528 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-360/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 22:50:21,887 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-360/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 22:50:21,897 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-360/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 22:50:21,913 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-360/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 22:50:21,915 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-360/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 22:50:30,499 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-285] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 304357.34 | bwd_microstep: 227.60 | bwd_inner_microstep: 227.25 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 164.16 | bwd_microstep: 227.90 | bwd_inner_microstep: 227.72 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 163.67 | bwd_microstep: 231.17 | bwd_inner_microstep: 230.92 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 164.55 | bwd_microstep: 254.42 | bwd_inner_microstep: 254.05 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 162.42 | bwd_microstep: 240.13 | bwd_inner_microstep: 240.02 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 164.78 | bwd_microstep: 237.26 | bwd_inner_microstep: 237.02 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 163.20 | bwd_microstep: 248.68 | bwd_inner_microstep: 248.51 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 187.74 | bwd_microstep: 257.54 | bwd_inner_microstep: 244.02 | bwd_allreduce_microstep: 13.38 | step_microstep: 21.50 [Rank 0] time (ms) | fwd: 305515.53 | bwd: 1924.83 | bwd_inner: 1909.85 | bwd_allreduce: 14.16 | step: 22.25 13%|█▎ | 361/2865 [2:29:51<69:57:38, 100.58s/it] {'loss': 0.8049, 'learning_rate': 3.9041289576449057e-05, 'epoch': 0.38} 13%|█▎ | 361/2865 [2:29:51<69:57:38, 100.58s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 162.61 | bwd_microstep: 223.25 | bwd_inner_microstep: 222.81 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 166.11 | bwd_microstep: 222.95 | bwd_inner_microstep: 222.81 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 166.64 | bwd_microstep: 223.96 | bwd_inner_microstep: 223.58 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 164.88 | bwd_microstep: 222.77 | bwd_inner_microstep: 222.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 164.00 | bwd_microstep: 227.42 | bwd_inner_microstep: 227.11 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 165.09 | bwd_microstep: 221.39 | bwd_inner_microstep: 221.14 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 164.38 | bwd_microstep: 222.49 | bwd_inner_microstep: 222.28 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.25 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 162.52 | bwd_microstep: 272.22 | bwd_inner_microstep: 259.56 | bwd_allreduce_microstep: 12.57 | step_microstep: 23.23 [Rank 0] time (ms) | fwd: 1316.13 | bwd: 1836.62 | bwd_inner: 1822.29 | bwd_allreduce: 13.41 | step: 24.02 13%|█▎ | 362/2865 [2:29:54<49:38:55, 71.41s/it] {'loss': 0.8073, 'learning_rate': 3.90343612173905e-05, 'epoch': 0.38} 13%|█▎ | 362/2865 [2:29:54<49:38:55, 71.41s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 944 [Rank 0] time (ms) | fwd_microstep: 164.06 | bwd_microstep: 224.46 | bwd_inner_microstep: 224.23 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 165.31 | bwd_microstep: 225.36 | bwd_inner_microstep: 225.22 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 163.39 | bwd_microstep: 221.97 | bwd_inner_microstep: 221.64 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 166.83 | bwd_microstep: 226.34 | bwd_inner_microstep: 226.22 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 165.28 | bwd_microstep: 223.38 | bwd_inner_microstep: 223.20 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 161.68 | bwd_microstep: 229.13 | bwd_inner_microstep: 228.90 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 163.25 | bwd_microstep: 220.95 | bwd_inner_microstep: 220.78 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 177.19 | bwd_microstep: 272.81 | bwd_inner_microstep: 255.95 | bwd_allreduce_microstep: 16.76 | step_microstep: 21.39 [Rank 0] time (ms) | fwd: 1326.90 | bwd: 1844.51 | bwd_inner: 1826.41 | bwd_allreduce: 17.36 | step: 21.94 13%|█▎ | 363/2865 [2:29:58<35:26:54, 51.01s/it] {'loss': 0.9288, 'learning_rate': 3.902740853288029e-05, 'epoch': 0.38} 13%|█▎ | 363/2865 [2:29:58<35:26:54, 51.01s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 165.93 | bwd_microstep: 217.11 | bwd_inner_microstep: 216.99 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 162.90 | bwd_microstep: 233.73 | bwd_inner_microstep: 233.47 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 164.34 | bwd_microstep: 228.66 | bwd_inner_microstep: 228.06 | bwd_allreduce_microstep: 0.33 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 164.54 | bwd_microstep: 234.09 | bwd_inner_microstep: 233.60 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 162.69 | bwd_microstep: 233.10 | bwd_inner_microstep: 232.74 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 166.41 | bwd_microstep: 239.39 | bwd_inner_microstep: 239.23 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 163.65 | bwd_microstep: 238.20 | bwd_inner_microstep: 237.98 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.96 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 168.92 | bwd_microstep: 268.16 | bwd_inner_microstep: 257.65 | bwd_allreduce_microstep: 10.41 | step_microstep: 21.36 [Rank 0] time (ms) | fwd: 1319.28 | bwd: 1892.62 | bwd_inner: 1880.18 | bwd_allreduce: 11.27 | step: 22.09 13%|█▎ | 364/2865 [2:30:01<25:30:42, 36.72s/it] {'loss': 0.9355, 'learning_rate': 3.9020431531803817e-05, 'epoch': 0.38} 13%|█▎ | 364/2865 [2:30:01<25:30:42, 36.72s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 169.56 | bwd_microstep: 227.90 | bwd_inner_microstep: 227.65 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 181.88 | bwd_microstep: 234.12 | bwd_inner_microstep: 233.86 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 165.79 | bwd_microstep: 231.55 | bwd_inner_microstep: 231.31 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 166.33 | bwd_microstep: 227.24 | bwd_inner_microstep: 227.05 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.53 | bwd_microstep: 230.23 | bwd_inner_microstep: 230.08 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 163.36 | bwd_microstep: 230.34 | bwd_inner_microstep: 230.13 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 167.26 | bwd_microstep: 230.06 | bwd_inner_microstep: 229.84 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 165.83 | bwd_microstep: 264.19 | bwd_inner_microstep: 254.43 | bwd_allreduce_microstep: 9.61 | step_microstep: 21.02 [Rank 0] time (ms) | fwd: 1343.39 | bwd: 1875.73 | bwd_inner: 1864.84 | bwd_allreduce: 10.14 | step: 21.63 13%|█▎ | 365/2865 [2:30:05<18:33:30, 26.72s/it] {'loss': 0.8972, 'learning_rate': 3.9013430223077514e-05, 'epoch': 0.38} 13%|█▎ | 365/2865 [2:30:05<18:33:30, 26.72s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 164.38 | bwd_microstep: 219.33 | bwd_inner_microstep: 219.14 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 173.52 | bwd_microstep: 231.84 | bwd_inner_microstep: 231.55 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 171.48 | bwd_microstep: 224.04 | bwd_inner_microstep: 223.67 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 166.86 | bwd_microstep: 220.65 | bwd_inner_microstep: 220.51 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 161.99 | bwd_microstep: 225.70 | bwd_inner_microstep: 225.56 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 163.97 | bwd_microstep: 223.57 | bwd_inner_microstep: 223.40 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 164.64 | bwd_microstep: 238.39 | bwd_inner_microstep: 238.01 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.95 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 163.30 | bwd_microstep: 260.89 | bwd_inner_microstep: 248.22 | bwd_allreduce_microstep: 12.55 | step_microstep: 23.76 [Rank 0] time (ms) | fwd: 1330.03 | bwd: 1844.59 | bwd_inner: 1830.41 | bwd_allreduce: 13.27 | step: 24.68 13%|█▎ | 366/2865 [2:30:08<13:41:01, 19.71s/it] {'loss': 0.9281, 'learning_rate': 3.9006404615648866e-05, 'epoch': 0.38} 13%|█▎ | 366/2865 [2:30:08<13:41:01, 19.71s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 161.78 | bwd_microstep: 232.32 | bwd_inner_microstep: 231.94 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 940 [Rank 0] time (ms) | fwd_microstep: 162.74 | bwd_microstep: 223.51 | bwd_inner_microstep: 223.38 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 162.63 | bwd_microstep: 222.44 | bwd_inner_microstep: 222.20 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 166.48 | bwd_microstep: 222.65 | bwd_inner_microstep: 222.52 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 163.73 | bwd_microstep: 234.06 | bwd_inner_microstep: 233.93 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.44 | bwd_microstep: 221.73 | bwd_inner_microstep: 221.63 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 162.34 | bwd_microstep: 227.32 | bwd_inner_microstep: 227.24 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.32 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.85 | bwd_microstep: 267.08 | bwd_inner_microstep: 250.89 | bwd_allreduce_microstep: 16.08 | step_microstep: 27.57 [Rank 0] time (ms) | fwd: 1303.87 | bwd: 1851.23 | bwd_inner: 1833.97 | bwd_allreduce: 16.63 | step: 28.22 13%|█▎ | 367/2865 [2:30:11<10:16:02, 14.80s/it] {'loss': 1.2577, 'learning_rate': 3.899935471849645e-05, 'epoch': 0.38} 13%|█▎ | 367/2865 [2:30:11<10:16:02, 14.80s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 165.29 | bwd_microstep: 234.92 | bwd_inner_microstep: 234.51 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 165.25 | bwd_microstep: 222.74 | bwd_inner_microstep: 222.40 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 169.33 | bwd_microstep: 220.59 | bwd_inner_microstep: 220.38 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.22 | bwd_microstep: 222.58 | bwd_inner_microstep: 222.43 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 164.53 | bwd_microstep: 218.31 | bwd_inner_microstep: 218.22 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 170.21 | bwd_microstep: 223.17 | bwd_inner_microstep: 223.03 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 163.81 | bwd_microstep: 223.81 | bwd_inner_microstep: 223.66 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.42 | bwd_microstep: 258.09 | bwd_inner_microstep: 247.59 | bwd_allreduce_microstep: 10.43 | step_microstep: 21.35 [Rank 0] time (ms) | fwd: 1324.93 | bwd: 1824.32 | bwd_inner: 1812.56 | bwd_allreduce: 10.98 | step: 21.99 13%|█▎ | 368/2865 [2:30:15<7:52:32, 11.35s/it] {'loss': 0.841, 'learning_rate': 3.899228054062985e-05, 'epoch': 0.38} 13%|█▎ | 368/2865 [2:30:15<7:52:32, 11.35s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 163.46 | bwd_microstep: 222.98 | bwd_inner_microstep: 222.87 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 187.93 | bwd_microstep: 221.03 | bwd_inner_microstep: 220.82 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 178.83 | bwd_microstep: 220.66 | bwd_inner_microstep: 220.53 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 146.74 | bwd_microstep: 199.19 | bwd_inner_microstep: 198.99 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 163.82 | bwd_microstep: 221.92 | bwd_inner_microstep: 221.71 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 161.85 | bwd_microstep: 225.42 | bwd_inner_microstep: 225.29 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 173.49 | bwd_microstep: 226.96 | bwd_inner_microstep: 226.57 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 162.05 | bwd_microstep: 258.72 | bwd_inner_microstep: 248.36 | bwd_allreduce_microstep: 10.29 | step_microstep: 21.58 [Rank 0] time (ms) | fwd: 1338.05 | bwd: 1797.02 | bwd_inner: 1785.36 | bwd_allreduce: 10.93 | step: 22.35 13%|█▎ | 369/2865 [2:30:18<6:12:03, 8.94s/it] {'loss': 0.8895, 'learning_rate': 3.89851820910897e-05, 'epoch': 0.39} 13%|█▎ | 369/2865 [2:30:18<6:12:03, 8.94s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 165.72 | bwd_microstep: 224.34 | bwd_inner_microstep: 224.21 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 164.10 | bwd_microstep: 227.05 | bwd_inner_microstep: 226.90 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 162.73 | bwd_microstep: 225.53 | bwd_inner_microstep: 225.45 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 150.35 | bwd_microstep: 199.60 | bwd_inner_microstep: 199.52 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 170.69 | bwd_microstep: 234.72 | bwd_inner_microstep: 234.39 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 162.95 | bwd_microstep: 223.60 | bwd_inner_microstep: 223.40 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 151.84 | bwd_microstep: 195.05 | bwd_inner_microstep: 194.77 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 162.64 | bwd_microstep: 259.54 | bwd_inner_microstep: 249.71 | bwd_allreduce_microstep: 9.75 | step_microstep: 21.31 [Rank 0] time (ms) | fwd: 1290.86 | bwd: 1789.57 | bwd_inner: 1778.53 | bwd_allreduce: 10.34 | step: 22.10 13%|█▎ | 370/2865 [2:30:21<5:00:52, 7.24s/it] {'loss': 0.8683, 'learning_rate': 3.897805937894764e-05, 'epoch': 0.39} 13%|█▎ | 370/2865 [2:30:21<5:00:52, 7.24s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 163.19 | bwd_microstep: 228.58 | bwd_inner_microstep: 228.26 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 164.93 | bwd_microstep: 243.73 | bwd_inner_microstep: 243.65 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 163.24 | bwd_microstep: 225.19 | bwd_inner_microstep: 225.08 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 162.89 | bwd_microstep: 223.14 | bwd_inner_microstep: 222.93 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 161.95 | bwd_microstep: 224.10 | bwd_inner_microstep: 223.93 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 177.30 | bwd_microstep: 223.03 | bwd_inner_microstep: 222.89 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 162.57 | bwd_microstep: 229.07 | bwd_inner_microstep: 228.90 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 162.72 | bwd_microstep: 259.76 | bwd_inner_microstep: 249.37 | bwd_allreduce_microstep: 10.31 | step_microstep: 21.43 [Rank 0] time (ms) | fwd: 1318.66 | bwd: 1856.70 | bwd_inner: 1845.20 | bwd_allreduce: 10.88 | step: 22.07 13%|█▎ | 371/2865 [2:30:24<4:12:07, 6.07s/it] {'loss': 0.9256, 'learning_rate': 3.897091241330633e-05, 'epoch': 0.39} 13%|█▎ | 371/2865 [2:30:24<4:12:07, 6.07s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 163.97 | bwd_microstep: 223.36 | bwd_inner_microstep: 223.12 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 164.96 | bwd_microstep: 242.17 | bwd_inner_microstep: 242.02 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 166.75 | bwd_microstep: 226.18 | bwd_inner_microstep: 225.84 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 163.21 | bwd_microstep: 236.60 | bwd_inner_microstep: 236.46 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 163.40 | bwd_microstep: 227.50 | bwd_inner_microstep: 227.29 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 164.02 | bwd_microstep: 227.71 | bwd_inner_microstep: 227.55 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 164.56 | bwd_microstep: 228.33 | bwd_inner_microstep: 228.00 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 166.61 | bwd_microstep: 270.71 | bwd_inner_microstep: 259.62 | bwd_allreduce_microstep: 11.00 | step_microstep: 21.08 [Rank 0] time (ms) | fwd: 1317.37 | bwd: 1882.67 | bwd_inner: 1870.22 | bwd_allreduce: 11.65 | step: 21.80 13%|█▎ | 372/2865 [2:30:28<3:38:34, 5.26s/it] {'loss': 0.9269, 'learning_rate': 3.896374120329942e-05, 'epoch': 0.39} 13%|█▎ | 372/2865 [2:30:28<3:38:34, 5.26s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 942 [Rank 0] time (ms) | fwd_microstep: 165.40 | bwd_microstep: 224.85 | bwd_inner_microstep: 224.72 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 163.43 | bwd_microstep: 232.13 | bwd_inner_microstep: 231.87 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 165.06 | bwd_microstep: 234.67 | bwd_inner_microstep: 234.46 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 164.02 | bwd_microstep: 235.95 | bwd_inner_microstep: 235.81 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.52 | bwd_microstep: 260.18 | bwd_inner_microstep: 260.00 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 172.21 | bwd_microstep: 242.35 | bwd_inner_microstep: 242.12 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 169.40 | bwd_microstep: 239.34 | bwd_inner_microstep: 239.15 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.01 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 165.39 | bwd_microstep: 276.49 | bwd_inner_microstep: 266.25 | bwd_allreduce_microstep: 10.14 | step_microstep: 22.17 [Rank 0] time (ms) | fwd: 1326.31 | bwd: 1946.08 | bwd_inner: 1934.63 | bwd_allreduce: 10.72 | step: 22.89 13%|█▎ | 373/2865 [2:30:31<3:16:02, 4.72s/it] {'loss': 0.8986, 'learning_rate': 3.895654575809153e-05, 'epoch': 0.39} 13%|█▎ | 373/2865 [2:30:31<3:16:02, 4.72s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 170.91 | bwd_microstep: 275.59 | bwd_inner_microstep: 275.42 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 197.92 | bwd_microstep: 256.56 | bwd_inner_microstep: 256.39 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 167.70 | bwd_microstep: 246.19 | bwd_inner_microstep: 245.99 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 170.74 | bwd_microstep: 265.27 | bwd_inner_microstep: 265.19 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 165.30 | bwd_microstep: 242.07 | bwd_inner_microstep: 241.88 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 172.57 | bwd_microstep: 227.96 | bwd_inner_microstep: 227.82 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 175.37 | bwd_microstep: 229.74 | bwd_inner_microstep: 229.65 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.50 | bwd_microstep: 260.73 | bwd_inner_microstep: 250.40 | bwd_allreduce_microstep: 10.25 | step_microstep: 21.14 [Rank 0] time (ms) | fwd: 1381.87 | bwd: 2004.23 | bwd_inner: 1992.92 | bwd_allreduce: 10.76 | step: 21.85 13%|█▎ | 374/2865 [2:30:35<3:01:44, 4.38s/it] {'loss': 0.9923, 'learning_rate': 3.894932608687827e-05, 'epoch': 0.39} 13%|█▎ | 374/2865 [2:30:35<3:01:44, 4.38s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 166.93 | bwd_microstep: 233.10 | bwd_inner_microstep: 232.92 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 167.15 | bwd_microstep: 236.92 | bwd_inner_microstep: 236.76 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 165.18 | bwd_microstep: 246.23 | bwd_inner_microstep: 246.09 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 169.30 | bwd_microstep: 241.56 | bwd_inner_microstep: 241.37 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 170.06 | bwd_microstep: 240.22 | bwd_inner_microstep: 239.88 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 168.06 | bwd_microstep: 251.96 | bwd_inner_microstep: 251.83 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 168.93 | bwd_microstep: 244.41 | bwd_inner_microstep: 244.21 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.94 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 170.16 | bwd_microstep: 328.99 | bwd_inner_microstep: 314.27 | bwd_allreduce_microstep: 14.60 | step_microstep: 21.32 [Rank 0] time (ms) | fwd: 1345.66 | bwd: 2023.52 | bwd_inner: 2007.65 | bwd_allreduce: 15.10 | step: 21.95 13%|█▎ | 375/2865 [2:30:38<2:51:20, 4.13s/it] {'loss': 0.8926, 'learning_rate': 3.8942082198886214e-05, 'epoch': 0.39} 13%|█▎ | 375/2865 [2:30:38<2:51:20, 4.13s/it][INFO|trainer.py:3242] 2026-05-14 22:51:21,568 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 22:51:21,568 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 22:51:21,568 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-375 [INFO|configuration_utils.py:473] 2026-05-14 22:56:28,976 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-375/config.json [INFO|configuration_utils.py:594] 2026-05-14 22:56:28,988 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-375/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 22:56:33,450 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-375/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 22:56:33,461 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-375/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 22:56:33,469 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-375/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 22:56:33,472 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-375/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 22:56:40,524 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-300] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 304249.62 | bwd_microstep: 243.43 | bwd_inner_microstep: 243.21 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 177.21 | bwd_microstep: 275.47 | bwd_inner_microstep: 275.31 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 179.18 | bwd_microstep: 252.29 | bwd_inner_microstep: 252.12 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 182.96 | bwd_microstep: 245.11 | bwd_inner_microstep: 244.98 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 175.49 | bwd_microstep: 244.00 | bwd_inner_microstep: 243.86 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 170.44 | bwd_microstep: 245.78 | bwd_inner_microstep: 245.69 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 167.49 | bwd_microstep: 256.91 | bwd_inner_microstep: 256.74 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.95 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 169.11 | bwd_microstep: 273.76 | bwd_inner_microstep: 263.40 | bwd_allreduce_microstep: 10.26 | step_microstep: 21.82 [Rank 0] time (ms) | fwd: 305458.89 | bwd: 2036.86 | bwd_inner: 2025.45 | bwd_allreduce: 10.83 | step: 22.41 13%|█▎ | 376/2865 [2:36:01<68:58:11, 99.76s/it] {'loss': 0.9404, 'learning_rate': 3.8934814103372874e-05, 'epoch': 0.39} 13%|█▎ | 376/2865 [2:36:01<68:58:11, 99.76s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 178.80 | bwd_microstep: 248.23 | bwd_inner_microstep: 247.95 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 175.18 | bwd_microstep: 279.74 | bwd_inner_microstep: 279.45 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 175.07 | bwd_microstep: 260.37 | bwd_inner_microstep: 260.11 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 170.93 | bwd_microstep: 243.10 | bwd_inner_microstep: 242.85 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 175.65 | bwd_microstep: 247.45 | bwd_inner_microstep: 247.26 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 176.26 | bwd_microstep: 246.32 | bwd_inner_microstep: 246.15 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 149.82 | bwd_microstep: 205.42 | bwd_inner_microstep: 205.22 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.96 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 165.62 | bwd_microstep: 285.77 | bwd_inner_microstep: 275.49 | bwd_allreduce_microstep: 10.19 | step_microstep: 21.90 [Rank 0] time (ms) | fwd: 1367.22 | bwd: 2016.54 | bwd_inner: 2004.84 | bwd_allreduce: 10.87 | step: 22.71 13%|█▎ | 377/2865 [2:36:05<49:00:12, 70.91s/it] {'loss': 0.8682, 'learning_rate': 3.892752180962668e-05, 'epoch': 0.39} 13%|█▎ | 377/2865 [2:36:05<49:00:12, 70.91s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 938 [Rank 0] time (ms) | fwd_microstep: 178.78 | bwd_microstep: 252.44 | bwd_inner_microstep: 252.27 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 169.62 | bwd_microstep: 246.60 | bwd_inner_microstep: 246.45 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 174.92 | bwd_microstep: 239.68 | bwd_inner_microstep: 239.54 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 167.88 | bwd_microstep: 234.19 | bwd_inner_microstep: 233.86 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 175.93 | bwd_microstep: 284.47 | bwd_inner_microstep: 284.10 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 169.50 | bwd_microstep: 247.46 | bwd_inner_microstep: 247.32 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 155.24 | bwd_microstep: 216.24 | bwd_inner_microstep: 216.08 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.93 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 172.36 | bwd_microstep: 292.94 | bwd_inner_microstep: 282.72 | bwd_allreduce_microstep: 10.14 | step_microstep: 21.17 [Rank 0] time (ms) | fwd: 1364.12 | bwd: 2014.11 | bwd_inner: 2002.67 | bwd_allreduce: 10.75 | step: 21.82 13%|█▎ | 378/2865 [2:36:08<35:01:33, 50.70s/it] {'loss': 0.9369, 'learning_rate': 3.8920205326967024e-05, 'epoch': 0.4} 13%|█▎ | 378/2865 [2:36:08<35:01:33, 50.70s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 943 [Rank 0] time (ms) | fwd_microstep: 195.37 | bwd_microstep: 251.62 | bwd_inner_microstep: 251.49 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 167.67 | bwd_microstep: 277.15 | bwd_inner_microstep: 276.87 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 182.86 | bwd_microstep: 256.59 | bwd_inner_microstep: 256.27 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 179.84 | bwd_microstep: 250.60 | bwd_inner_microstep: 250.29 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 171.81 | bwd_microstep: 253.37 | bwd_inner_microstep: 253.21 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 168.20 | bwd_microstep: 249.01 | bwd_inner_microstep: 248.68 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 171.87 | bwd_microstep: 253.53 | bwd_inner_microstep: 253.34 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 147.97 | bwd_microstep: 255.26 | bwd_inner_microstep: 245.10 | bwd_allreduce_microstep: 10.07 | step_microstep: 21.57 [Rank 0] time (ms) | fwd: 1385.48 | bwd: 2047.28 | bwd_inner: 2035.61 | bwd_allreduce: 10.82 | step: 22.45 13%|█▎ | 379/2865 [2:36:12<25:15:32, 36.58s/it] {'loss': 0.9146, 'learning_rate': 3.89128646647442e-05, 'epoch': 0.4} 13%|█▎ | 379/2865 [2:36:12<25:15:32, 36.58s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 172.58 | bwd_microstep: 261.01 | bwd_inner_microstep: 260.81 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 174.97 | bwd_microstep: 243.23 | bwd_inner_microstep: 243.03 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 169.08 | bwd_microstep: 249.36 | bwd_inner_microstep: 249.19 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 168.10 | bwd_microstep: 257.16 | bwd_inner_microstep: 256.99 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 172.17 | bwd_microstep: 241.47 | bwd_inner_microstep: 241.25 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 167.76 | bwd_microstep: 249.96 | bwd_inner_microstep: 249.86 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 168.36 | bwd_microstep: 251.94 | bwd_inner_microstep: 251.73 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 169.35 | bwd_microstep: 288.95 | bwd_inner_microstep: 278.86 | bwd_allreduce_microstep: 10.00 | step_microstep: 21.74 [Rank 0] time (ms) | fwd: 1362.29 | bwd: 2043.19 | bwd_inner: 2031.95 | bwd_allreduce: 10.55 | step: 22.42 13%|█▎ | 380/2865 [2:36:16<18:25:26, 26.69s/it] {'loss': 0.8491, 'learning_rate': 3.890549983233938e-05, 'epoch': 0.4} 13%|█▎ | 380/2865 [2:36:16<18:25:26, 26.69s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 169.82 | bwd_microstep: 247.84 | bwd_inner_microstep: 247.54 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 168.30 | bwd_microstep: 242.30 | bwd_inner_microstep: 242.17 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 176.82 | bwd_microstep: 241.64 | bwd_inner_microstep: 241.46 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 166.58 | bwd_microstep: 237.39 | bwd_inner_microstep: 237.23 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 165.94 | bwd_microstep: 229.60 | bwd_inner_microstep: 229.45 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 166.87 | bwd_microstep: 254.82 | bwd_inner_microstep: 254.41 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 172.79 | bwd_microstep: 231.52 | bwd_inner_microstep: 231.15 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.36 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 168.58 | bwd_microstep: 286.32 | bwd_inner_microstep: 270.01 | bwd_allreduce_microstep: 16.19 | step_microstep: 27.98 [Rank 0] time (ms) | fwd: 1355.60 | bwd: 1971.61 | bwd_inner: 1953.76 | bwd_allreduce: 16.93 | step: 28.81 13%|█▎ | 381/2865 [2:36:19<13:37:12, 19.74s/it] {'loss': 0.6624, 'learning_rate': 3.889811083916464e-05, 'epoch': 0.4} 13%|█▎ | 381/2865 [2:36:19<13:37:12, 19.74s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 176.50 | bwd_microstep: 229.56 | bwd_inner_microstep: 229.43 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 169.80 | bwd_microstep: 233.50 | bwd_inner_microstep: 233.28 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 167.74 | bwd_microstep: 242.49 | bwd_inner_microstep: 242.36 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 169.40 | bwd_microstep: 244.29 | bwd_inner_microstep: 244.07 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 163.40 | bwd_microstep: 237.14 | bwd_inner_microstep: 237.00 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 173.02 | bwd_microstep: 232.59 | bwd_inner_microstep: 232.33 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 164.12 | bwd_microstep: 248.93 | bwd_inner_microstep: 248.79 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.37 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.06 | bwd_microstep: 276.19 | bwd_inner_microstep: 266.34 | bwd_allreduce_microstep: 9.67 | step_microstep: 23.44 [Rank 0] time (ms) | fwd: 1344.92 | bwd: 1944.79 | bwd_inner: 1933.95 | bwd_allreduce: 10.16 | step: 24.03 13%|█▎ | 382/2865 [2:36:23<10:14:57, 14.86s/it] {'loss': 0.8896, 'learning_rate': 3.8890697694662944e-05, 'epoch': 0.4} 13%|█▎ | 382/2865 [2:36:23<10:14:57, 14.86s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 163.45 | bwd_microstep: 241.46 | bwd_inner_microstep: 241.26 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 170.38 | bwd_microstep: 243.44 | bwd_inner_microstep: 243.35 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 164.16 | bwd_microstep: 240.11 | bwd_inner_microstep: 240.04 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 163.12 | bwd_microstep: 243.55 | bwd_inner_microstep: 243.34 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 176.69 | bwd_microstep: 234.92 | bwd_inner_microstep: 234.83 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 163.40 | bwd_microstep: 247.17 | bwd_inner_microstep: 246.93 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 170.36 | bwd_microstep: 216.21 | bwd_inner_microstep: 216.13 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 167.58 | bwd_microstep: 272.40 | bwd_inner_microstep: 261.08 | bwd_allreduce_microstep: 11.15 | step_microstep: 21.38 [Rank 0] time (ms) | fwd: 1339.04 | bwd: 1939.34 | bwd_inner: 1927.14 | bwd_allreduce: 11.57 | step: 21.96 13%|█▎ | 383/2865 [2:36:26<7:53:21, 11.44s/it] {'loss': 0.835, 'learning_rate': 3.888326040830811e-05, 'epoch': 0.4} 13%|█▎ | 383/2865 [2:36:26<7:53:21, 11.44s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 166.54 | bwd_microstep: 226.43 | bwd_inner_microstep: 226.23 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 163.75 | bwd_microstep: 228.79 | bwd_inner_microstep: 228.62 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 167.44 | bwd_microstep: 237.97 | bwd_inner_microstep: 237.61 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 180.88 | bwd_microstep: 230.08 | bwd_inner_microstep: 229.87 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.67 | bwd_microstep: 242.75 | bwd_inner_microstep: 242.57 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 166.42 | bwd_microstep: 227.11 | bwd_inner_microstep: 226.92 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 151.89 | bwd_microstep: 206.52 | bwd_inner_microstep: 206.34 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.95 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 165.94 | bwd_microstep: 297.10 | bwd_inner_microstep: 283.77 | bwd_allreduce_microstep: 13.23 | step_microstep: 23.39 [Rank 0] time (ms) | fwd: 1326.42 | bwd: 1896.88 | bwd_inner: 1882.32 | bwd_allreduce: 13.85 | step: 24.19 13%|█▎ | 384/2865 [2:36:30<6:13:29, 9.03s/it] {'loss': 1.0182, 'learning_rate': 3.88757989896048e-05, 'epoch': 0.4} 13%|█▎ | 384/2865 [2:36:30<6:13:29, 9.03s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 947 [Rank 0] time (ms) | fwd_microstep: 167.85 | bwd_microstep: 228.33 | bwd_inner_microstep: 228.21 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 162.76 | bwd_microstep: 231.32 | bwd_inner_microstep: 231.19 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 162.07 | bwd_microstep: 233.14 | bwd_inner_microstep: 233.04 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 162.07 | bwd_microstep: 240.35 | bwd_inner_microstep: 240.18 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 163.68 | bwd_microstep: 235.93 | bwd_inner_microstep: 235.85 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 166.30 | bwd_microstep: 240.51 | bwd_inner_microstep: 240.33 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 163.95 | bwd_microstep: 230.25 | bwd_inner_microstep: 229.97 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.93 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 165.86 | bwd_microstep: 279.88 | bwd_inner_microstep: 269.74 | bwd_allreduce_microstep: 10.03 | step_microstep: 22.01 [Rank 0] time (ms) | fwd: 1314.42 | bwd: 1919.82 | bwd_inner: 1908.65 | bwd_allreduce: 10.53 | step: 22.77 13%|█▎ | 385/2865 [2:36:33<5:03:40, 7.35s/it] {'loss': 1.0434, 'learning_rate': 3.8868313448088544e-05, 'epoch': 0.4} 13%|█▎ | 385/2865 [2:36:33<5:03:40, 7.35s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 943 [Rank 0] time (ms) | fwd_microstep: 168.36 | bwd_microstep: 272.59 | bwd_inner_microstep: 272.27 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 172.08 | bwd_microstep: 237.89 | bwd_inner_microstep: 237.33 | bwd_allreduce_microstep: 0.22 | step_microstep: 0.27 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 164.75 | bwd_microstep: 243.99 | bwd_inner_microstep: 243.80 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 163.73 | bwd_microstep: 231.97 | bwd_inner_microstep: 231.76 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 168.11 | bwd_microstep: 236.86 | bwd_inner_microstep: 236.63 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 174.35 | bwd_microstep: 240.24 | bwd_inner_microstep: 239.94 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 167.37 | bwd_microstep: 252.85 | bwd_inner_microstep: 252.61 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.04 | optimizer_step: 4.86 [Rank 0] time (ms) | fwd_microstep: 167.85 | bwd_microstep: 270.32 | bwd_inner_microstep: 255.07 | bwd_allreduce_microstep: 15.06 | step_microstep: 23.87 [Rank 0] time (ms) | fwd: 1346.49 | bwd: 1986.86 | bwd_inner: 1969.90 | bwd_allreduce: 15.90 | step: 24.68 13%|█▎ | 386/2865 [2:36:36<4:16:10, 6.20s/it] {'loss': 1.0036, 'learning_rate': 3.8860803793325665e-05, 'epoch': 0.4} 13%|█▎ | 386/2865 [2:36:36<4:16:10, 6.20s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 168.85 | bwd_microstep: 237.31 | bwd_inner_microstep: 237.04 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 163.90 | bwd_microstep: 252.23 | bwd_inner_microstep: 251.82 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 177.51 | bwd_microstep: 296.41 | bwd_inner_microstep: 296.12 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 163.81 | bwd_microstep: 244.66 | bwd_inner_microstep: 244.28 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 163.15 | bwd_microstep: 230.79 | bwd_inner_microstep: 230.49 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 167.90 | bwd_microstep: 248.84 | bwd_inner_microstep: 248.71 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 169.85 | bwd_microstep: 238.49 | bwd_inner_microstep: 238.33 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 187.30 | bwd_microstep: 269.16 | bwd_inner_microstep: 258.35 | bwd_allreduce_microstep: 10.66 | step_microstep: 21.01 [Rank 0] time (ms) | fwd: 1362.18 | bwd: 2018.06 | bwd_inner: 2005.55 | bwd_allreduce: 11.48 | step: 22.01 14%|█▎ | 387/2865 [2:36:40<3:43:26, 5.41s/it] {'loss': 0.8763, 'learning_rate': 3.885327003491332e-05, 'epoch': 0.4} 14%|█▎ | 387/2865 [2:36:40<3:43:26, 5.41s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 163.87 | bwd_microstep: 233.32 | bwd_inner_microstep: 232.95 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 162.43 | bwd_microstep: 235.95 | bwd_inner_microstep: 235.73 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 170.36 | bwd_microstep: 254.04 | bwd_inner_microstep: 253.90 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 166.00 | bwd_microstep: 226.40 | bwd_inner_microstep: 226.24 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 165.31 | bwd_microstep: 224.65 | bwd_inner_microstep: 224.49 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 170.65 | bwd_microstep: 222.19 | bwd_inner_microstep: 221.98 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 164.74 | bwd_microstep: 222.64 | bwd_inner_microstep: 222.51 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.60 | bwd_microstep: 255.81 | bwd_inner_microstep: 245.90 | bwd_allreduce_microstep: 9.75 | step_microstep: 21.05 [Rank 0] time (ms) | fwd: 1326.83 | bwd: 1875.11 | bwd_inner: 1864.04 | bwd_allreduce: 10.32 | step: 21.62 14%|█▎ | 388/2865 [2:36:43<3:18:05, 4.80s/it] {'loss': 1.0394, 'learning_rate': 3.8845712182479485e-05, 'epoch': 0.41} 14%|█▎ | 388/2865 [2:36:43<3:18:05, 4.80s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 161.75 | bwd_microstep: 219.39 | bwd_inner_microstep: 219.04 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 174.58 | bwd_microstep: 232.29 | bwd_inner_microstep: 232.11 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 169.91 | bwd_microstep: 228.06 | bwd_inner_microstep: 227.83 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 171.41 | bwd_microstep: 226.53 | bwd_inner_microstep: 226.15 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.94 | bwd_microstep: 229.71 | bwd_inner_microstep: 229.54 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.19 | bwd_microstep: 247.04 | bwd_inner_microstep: 246.88 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 166.56 | bwd_microstep: 222.92 | bwd_inner_microstep: 222.52 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 165.46 | bwd_microstep: 257.74 | bwd_inner_microstep: 247.32 | bwd_allreduce_microstep: 10.30 | step_microstep: 21.20 [Rank 0] time (ms) | fwd: 1331.67 | bwd: 1863.87 | bwd_inner: 1851.79 | bwd_allreduce: 11.07 | step: 22.22 14%|█▎ | 389/2865 [2:36:47<3:00:26, 4.37s/it] {'loss': 1.0175, 'learning_rate': 3.88381302456829e-05, 'epoch': 0.41} 14%|█▎ | 389/2865 [2:36:47<3:00:26, 4.37s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 164.48 | bwd_microstep: 219.19 | bwd_inner_microstep: 218.86 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.98 | bwd_microstep: 225.43 | bwd_inner_microstep: 225.10 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 166.55 | bwd_microstep: 231.44 | bwd_inner_microstep: 231.27 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 149.23 | bwd_microstep: 200.77 | bwd_inner_microstep: 200.55 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 168.70 | bwd_microstep: 255.83 | bwd_inner_microstep: 255.53 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 162.14 | bwd_microstep: 220.19 | bwd_inner_microstep: 219.87 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 161.43 | bwd_microstep: 222.14 | bwd_inner_microstep: 221.95 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.97 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.86 | bwd_microstep: 260.23 | bwd_inner_microstep: 249.85 | bwd_allreduce_microstep: 10.28 | step_microstep: 21.23 [Rank 0] time (ms) | fwd: 1300.24 | bwd: 1835.34 | bwd_inner: 1823.36 | bwd_allreduce: 11.08 | step: 21.90 14%|█▎ | 390/2865 [2:36:50<2:47:21, 4.06s/it] {'loss': 0.8687, 'learning_rate': 3.883052423421309e-05, 'epoch': 0.41} 14%|█▎ | 390/2865 [2:36:50<2:47:21, 4.06s/it][INFO|trainer.py:3242] 2026-05-14 22:57:33,289 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 22:57:33,290 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 22:57:33,290 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-390 [INFO|configuration_utils.py:473] 2026-05-14 23:02:41,236 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-390/config.json [INFO|configuration_utils.py:594] 2026-05-14 23:02:41,241 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-390/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 23:02:45,771 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-390/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 23:02:45,779 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-390/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 23:02:45,786 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-390/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 23:02:45,787 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-390/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 23:02:52,484 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-315] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 942 [Rank 0] time (ms) | fwd_microstep: 304589.96 | bwd_microstep: 228.63 | bwd_inner_microstep: 228.47 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 162.61 | bwd_microstep: 231.39 | bwd_inner_microstep: 231.17 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 164.08 | bwd_microstep: 231.59 | bwd_inner_microstep: 231.38 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 165.29 | bwd_microstep: 230.41 | bwd_inner_microstep: 230.28 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 164.91 | bwd_microstep: 222.16 | bwd_inner_microstep: 222.08 | bwd_allreduce_microstep: 0.03 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.69 | bwd_microstep: 220.56 | bwd_inner_microstep: 220.48 | bwd_allreduce_microstep: 0.03 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 162.34 | bwd_microstep: 226.23 | bwd_inner_microstep: 225.92 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 161.47 | bwd_microstep: 262.36 | bwd_inner_microstep: 247.34 | bwd_allreduce_microstep: 14.94 | step_microstep: 22.09 [Rank 0] time (ms) | fwd: 305719.69 | bwd: 1853.43 | bwd_inner: 1837.35 | bwd_allreduce: 15.47 | step: 22.84 14%|█▎ | 391/2865 [2:42:13<68:29:35, 99.67s/it] {'loss': 1.248, 'learning_rate': 3.8822894157790374e-05, 'epoch': 0.41} 14%|█▎ | 391/2865 [2:42:13<68:29:35, 99.67s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 945 [Rank 0] time (ms) | fwd_microstep: 163.93 | bwd_microstep: 222.93 | bwd_inner_microstep: 222.78 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 161.39 | bwd_microstep: 225.83 | bwd_inner_microstep: 225.68 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 162.83 | bwd_microstep: 232.94 | bwd_inner_microstep: 232.78 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 163.24 | bwd_microstep: 229.02 | bwd_inner_microstep: 228.92 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 161.61 | bwd_microstep: 312.20 | bwd_inner_microstep: 312.09 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 178.58 | bwd_microstep: 236.61 | bwd_inner_microstep: 236.24 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.11 | bwd_microstep: 262.64 | bwd_inner_microstep: 262.48 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.11 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 164.09 | bwd_microstep: 275.15 | bwd_inner_microstep: 263.96 | bwd_allreduce_microstep: 10.98 | step_microstep: 21.74 [Rank 0] time (ms) | fwd: 1318.65 | bwd: 1997.43 | bwd_inner: 1985.16 | bwd_allreduce: 11.50 | step: 22.41 14%|█▎ | 392/2865 [2:42:16<48:38:44, 70.81s/it] {'loss': 0.9942, 'learning_rate': 3.881524002616578e-05, 'epoch': 0.41} 14%|█▎ | 392/2865 [2:42:16<48:38:44, 70.81s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 168.95 | bwd_microstep: 245.88 | bwd_inner_microstep: 245.67 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 170.34 | bwd_microstep: 242.98 | bwd_inner_microstep: 242.70 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 177.75 | bwd_microstep: 235.23 | bwd_inner_microstep: 234.87 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 163.27 | bwd_microstep: 241.70 | bwd_inner_microstep: 241.62 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 163.08 | bwd_microstep: 246.11 | bwd_inner_microstep: 245.97 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 165.37 | bwd_microstep: 230.38 | bwd_inner_microstep: 230.30 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 166.33 | bwd_microstep: 247.00 | bwd_inner_microstep: 246.91 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.27 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.58 | bwd_microstep: 271.99 | bwd_inner_microstep: 259.23 | bwd_allreduce_microstep: 12.67 | step_microstep: 23.81 [Rank 0] time (ms) | fwd: 1338.53 | bwd: 1961.40 | bwd_inner: 1947.45 | bwd_allreduce: 13.26 | step: 24.69 14%|█▎ | 393/2865 [2:42:20<34:45:21, 50.62s/it] {'loss': 0.8235, 'learning_rate': 3.8807561849121125e-05, 'epoch': 0.41} 14%|█▎ | 393/2865 [2:42:20<34:45:21, 50.62s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 164.74 | bwd_microstep: 225.80 | bwd_inner_microstep: 225.54 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 163.93 | bwd_microstep: 219.74 | bwd_inner_microstep: 219.63 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 162.59 | bwd_microstep: 254.22 | bwd_inner_microstep: 254.10 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.64 | bwd_microstep: 223.83 | bwd_inner_microstep: 223.74 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 180.02 | bwd_microstep: 231.45 | bwd_inner_microstep: 231.11 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 164.73 | bwd_microstep: 256.99 | bwd_inner_microstep: 256.87 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 163.68 | bwd_microstep: 252.79 | bwd_inner_microstep: 252.45 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.34 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 165.82 | bwd_microstep: 317.26 | bwd_inner_microstep: 303.74 | bwd_allreduce_microstep: 13.34 | step_microstep: 24.98 [Rank 0] time (ms) | fwd: 1327.02 | bwd: 1982.23 | bwd_inner: 1967.44 | bwd_allreduce: 13.94 | step: 25.75 14%|█▍ | 394/2865 [2:42:23<25:02:15, 36.48s/it] {'loss': 0.9591, 'learning_rate': 3.8799859636468914e-05, 'epoch': 0.41} 14%|█▍ | 394/2865 [2:42:23<25:02:15, 36.48s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 171.24 | bwd_microstep: 251.22 | bwd_inner_microstep: 251.02 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 180.69 | bwd_microstep: 247.67 | bwd_inner_microstep: 247.55 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 170.74 | bwd_microstep: 275.30 | bwd_inner_microstep: 275.17 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 175.09 | bwd_microstep: 244.24 | bwd_inner_microstep: 244.05 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 181.24 | bwd_microstep: 235.11 | bwd_inner_microstep: 234.98 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 168.18 | bwd_microstep: 237.52 | bwd_inner_microstep: 237.33 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 165.38 | bwd_microstep: 284.05 | bwd_inner_microstep: 283.82 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.98 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 175.54 | bwd_microstep: 311.59 | bwd_inner_microstep: 300.40 | bwd_allreduce_microstep: 11.04 | step_microstep: 21.48 [Rank 0] time (ms) | fwd: 1387.97 | bwd: 2086.81 | bwd_inner: 2074.63 | bwd_allreduce: 11.54 | step: 22.14 14%|█▍ | 395/2865 [2:42:27<18:16:17, 26.63s/it] {'loss': 1.0208, 'learning_rate': 3.8792133398052394e-05, 'epoch': 0.41} 14%|█▍ | 395/2865 [2:42:27<18:16:17, 26.63s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 183.75 | bwd_microstep: 224.67 | bwd_inner_microstep: 224.53 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 163.42 | bwd_microstep: 225.70 | bwd_inner_microstep: 225.56 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 165.45 | bwd_microstep: 233.96 | bwd_inner_microstep: 233.77 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 166.07 | bwd_microstep: 246.75 | bwd_inner_microstep: 246.60 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 165.82 | bwd_microstep: 241.33 | bwd_inner_microstep: 241.25 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 171.29 | bwd_microstep: 235.97 | bwd_inner_microstep: 235.90 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 162.89 | bwd_microstep: 233.73 | bwd_inner_microstep: 233.66 | bwd_allreduce_microstep: 0.03 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 178.01 | bwd_microstep: 260.51 | bwd_inner_microstep: 250.45 | bwd_allreduce_microstep: 9.96 | step_microstep: 21.11 [Rank 0] time (ms) | fwd: 1356.58 | bwd: 1902.70 | bwd_inner: 1891.85 | bwd_allreduce: 10.36 | step: 21.75 14%|█▍ | 396/2865 [2:42:30<13:29:24, 19.67s/it] {'loss': 1.0628, 'learning_rate': 3.878438314374552e-05, 'epoch': 0.41} 14%|█▍ | 396/2865 [2:42:30<13:29:24, 19.67s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 162.61 | bwd_microstep: 222.81 | bwd_inner_microstep: 222.51 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 164.50 | bwd_microstep: 238.63 | bwd_inner_microstep: 238.44 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 163.95 | bwd_microstep: 233.58 | bwd_inner_microstep: 233.36 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 168.76 | bwd_microstep: 235.02 | bwd_inner_microstep: 234.72 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.79 | bwd_microstep: 241.26 | bwd_inner_microstep: 241.01 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 162.51 | bwd_microstep: 210.28 | bwd_inner_microstep: 210.14 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 165.14 | bwd_microstep: 239.31 | bwd_inner_microstep: 239.18 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.10 | bwd_microstep: 278.30 | bwd_inner_microstep: 268.58 | bwd_allreduce_microstep: 9.65 | step_microstep: 21.20 [Rank 0] time (ms) | fwd: 1312.23 | bwd: 1899.32 | bwd_inner: 1888.28 | bwd_allreduce: 10.27 | step: 21.83 14%|█▍ | 397/2865 [2:42:34<10:08:01, 14.78s/it] {'loss': 0.9475, 'learning_rate': 3.877660888345292e-05, 'epoch': 0.42} 14%|█▍ | 397/2865 [2:42:34<10:08:01, 14.78s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 944 [Rank 0] time (ms) | fwd_microstep: 163.77 | bwd_microstep: 234.26 | bwd_inner_microstep: 234.06 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 165.45 | bwd_microstep: 235.19 | bwd_inner_microstep: 235.06 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 163.57 | bwd_microstep: 237.72 | bwd_inner_microstep: 237.55 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 162.52 | bwd_microstep: 234.54 | bwd_inner_microstep: 234.23 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 162.19 | bwd_microstep: 224.70 | bwd_inner_microstep: 224.58 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 161.96 | bwd_microstep: 226.37 | bwd_inner_microstep: 226.21 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 164.14 | bwd_microstep: 225.68 | bwd_inner_microstep: 225.36 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.93 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 148.62 | bwd_microstep: 231.27 | bwd_inner_microstep: 220.77 | bwd_allreduce_microstep: 10.42 | step_microstep: 21.16 [Rank 0] time (ms) | fwd: 1292.12 | bwd: 1849.82 | bwd_inner: 1838.16 | bwd_allreduce: 10.98 | step: 21.70 14%|█▍ | 398/2865 [2:42:37<7:46:10, 11.34s/it] {'loss': 0.8035, 'learning_rate': 3.8768810627109904e-05, 'epoch': 0.42} 14%|█▍ | 398/2865 [2:42:37<7:46:10, 11.34s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 166.46 | bwd_microstep: 224.41 | bwd_inner_microstep: 224.28 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 170.82 | bwd_microstep: 223.81 | bwd_inner_microstep: 223.59 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 167.46 | bwd_microstep: 222.79 | bwd_inner_microstep: 222.62 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 169.95 | bwd_microstep: 233.00 | bwd_inner_microstep: 232.71 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 166.08 | bwd_microstep: 241.25 | bwd_inner_microstep: 240.93 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 163.91 | bwd_microstep: 226.12 | bwd_inner_microstep: 225.81 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 165.83 | bwd_microstep: 240.42 | bwd_inner_microstep: 240.28 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 167.47 | bwd_microstep: 265.99 | bwd_inner_microstep: 255.18 | bwd_allreduce_microstep: 10.68 | step_microstep: 21.25 [Rank 0] time (ms) | fwd: 1337.85 | bwd: 1877.90 | bwd_inner: 1865.79 | bwd_allreduce: 11.31 | step: 21.89 14%|█▍ | 399/2865 [2:42:40<6:08:01, 8.95s/it] {'loss': 0.9432, 'learning_rate': 3.8760988384682464e-05, 'epoch': 0.42} 14%|█▍ | 399/2865 [2:42:40<6:08:01, 8.95s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 165.56 | bwd_microstep: 226.40 | bwd_inner_microstep: 226.18 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 173.24 | bwd_microstep: 227.10 | bwd_inner_microstep: 226.92 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 167.18 | bwd_microstep: 223.60 | bwd_inner_microstep: 223.46 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 166.25 | bwd_microstep: 216.95 | bwd_inner_microstep: 216.78 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 168.18 | bwd_microstep: 218.54 | bwd_inner_microstep: 218.40 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 151.40 | bwd_microstep: 196.26 | bwd_inner_microstep: 195.80 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 164.85 | bwd_microstep: 228.72 | bwd_inner_microstep: 228.56 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.86 [Rank 0] time (ms) | fwd_microstep: 166.22 | bwd_microstep: 255.86 | bwd_inner_microstep: 244.48 | bwd_allreduce_microstep: 11.26 | step_microstep: 21.19 [Rank 0] time (ms) | fwd: 1322.75 | bwd: 1793.57 | bwd_inner: 1780.88 | bwd_allreduce: 11.92 | step: 21.75 14%|█▍ | 400/2865 [2:42:44<4:58:03, 7.26s/it] {'loss': 0.8064, 'learning_rate': 3.875314216616723e-05, 'epoch': 0.42} 14%|█▍ | 400/2865 [2:42:44<4:58:03, 7.26s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 938 [Rank 0] time (ms) | fwd_microstep: 161.94 | bwd_microstep: 223.29 | bwd_inner_microstep: 223.12 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 224.91 | bwd_microstep: 221.20 | bwd_inner_microstep: 220.94 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 164.96 | bwd_microstep: 221.97 | bwd_inner_microstep: 221.74 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.20 | bwd_microstep: 217.83 | bwd_inner_microstep: 217.70 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 162.52 | bwd_microstep: 222.95 | bwd_inner_microstep: 222.78 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.24 | bwd_microstep: 223.98 | bwd_inner_microstep: 223.74 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 162.33 | bwd_microstep: 223.52 | bwd_inner_microstep: 223.27 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.70 | bwd_microstep: 253.58 | bwd_inner_microstep: 243.26 | bwd_allreduce_microstep: 10.20 | step_microstep: 21.03 [Rank 0] time (ms) | fwd: 1359.69 | bwd: 1808.42 | bwd_inner: 1796.99 | bwd_allreduce: 10.69 | step: 21.63 14%|█▍ | 401/2865 [2:42:47<4:09:34, 6.08s/it] {'loss': 0.8382, 'learning_rate': 3.874527198159149e-05, 'epoch': 0.42} 14%|█▍ | 401/2865 [2:42:47<4:09:34, 6.08s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 164.47 | bwd_microstep: 221.32 | bwd_inner_microstep: 220.93 | bwd_allreduce_microstep: 0.20 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 162.56 | bwd_microstep: 219.30 | bwd_inner_microstep: 218.93 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 162.02 | bwd_microstep: 218.53 | bwd_inner_microstep: 218.31 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 163.44 | bwd_microstep: 217.72 | bwd_inner_microstep: 217.43 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 186.95 | bwd_microstep: 217.27 | bwd_inner_microstep: 217.13 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 163.86 | bwd_microstep: 220.21 | bwd_inner_microstep: 220.07 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 164.67 | bwd_microstep: 217.28 | bwd_inner_microstep: 216.90 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 166.40 | bwd_microstep: 254.49 | bwd_inner_microstep: 244.08 | bwd_allreduce_microstep: 10.34 | step_microstep: 21.40 [Rank 0] time (ms) | fwd: 1334.26 | bwd: 1786.28 | bwd_inner: 1774.20 | bwd_allreduce: 11.18 | step: 22.30 14%|█▍ | 402/2865 [2:42:50<3:35:17, 5.24s/it] {'loss': 0.8726, 'learning_rate': 3.8737377841013135e-05, 'epoch': 0.42} 14%|█▍ | 402/2865 [2:42:50<3:35:17, 5.24s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 166.19 | bwd_microstep: 217.08 | bwd_inner_microstep: 216.96 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 165.15 | bwd_microstep: 230.23 | bwd_inner_microstep: 230.03 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 163.15 | bwd_microstep: 240.10 | bwd_inner_microstep: 239.87 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 179.56 | bwd_microstep: 238.07 | bwd_inner_microstep: 237.91 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 163.11 | bwd_microstep: 235.38 | bwd_inner_microstep: 235.12 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 173.67 | bwd_microstep: 244.19 | bwd_inner_microstep: 243.91 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 169.74 | bwd_microstep: 239.83 | bwd_inner_microstep: 239.69 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.97 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 165.80 | bwd_microstep: 276.29 | bwd_inner_microstep: 265.89 | bwd_allreduce_microstep: 10.32 | step_microstep: 23.74 [Rank 0] time (ms) | fwd: 1346.22 | bwd: 1921.26 | bwd_inner: 1909.67 | bwd_allreduce: 10.89 | step: 24.40 14%|█▍ | 403/2865 [2:42:54<3:13:09, 4.71s/it] {'loss': 0.8602, 'learning_rate': 3.872945975452069e-05, 'epoch': 0.42} 14%|█▍ | 403/2865 [2:42:54<3:13:09, 4.71s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 162.12 | bwd_microstep: 224.49 | bwd_inner_microstep: 224.38 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 165.62 | bwd_microstep: 258.00 | bwd_inner_microstep: 257.80 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 161.33 | bwd_microstep: 268.25 | bwd_inner_microstep: 267.86 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 162.87 | bwd_microstep: 269.86 | bwd_inner_microstep: 269.60 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 160.83 | bwd_microstep: 222.04 | bwd_inner_microstep: 221.90 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.62 | bwd_microstep: 225.74 | bwd_inner_microstep: 225.38 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.70 | bwd_microstep: 224.34 | bwd_inner_microstep: 223.95 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 164.87 | bwd_microstep: 256.00 | bwd_inner_microstep: 245.37 | bwd_allreduce_microstep: 10.46 | step_microstep: 21.35 [Rank 0] time (ms) | fwd: 1301.88 | bwd: 1948.91 | bwd_inner: 1936.61 | bwd_allreduce: 11.25 | step: 22.30 14%|█▍ | 404/2865 [2:42:57<2:57:02, 4.32s/it] {'loss': 0.8524, 'learning_rate': 3.872151773223328e-05, 'epoch': 0.42} 14%|█▍ | 404/2865 [2:42:57<2:57:02, 4.32s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.94 | bwd_microstep: 225.75 | bwd_inner_microstep: 225.46 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 160.17 | bwd_microstep: 224.87 | bwd_inner_microstep: 224.74 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.48 | bwd_microstep: 219.40 | bwd_inner_microstep: 219.28 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 161.29 | bwd_microstep: 221.34 | bwd_inner_microstep: 221.08 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 162.66 | bwd_microstep: 226.34 | bwd_inner_microstep: 226.16 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.54 | bwd_microstep: 216.62 | bwd_inner_microstep: 216.32 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.88 | bwd_microstep: 219.34 | bwd_inner_microstep: 219.11 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.24 | bwd_microstep: 257.31 | bwd_inner_microstep: 247.64 | bwd_allreduce_microstep: 9.58 | step_microstep: 21.25 [Rank 0] time (ms) | fwd: 1292.09 | bwd: 1811.09 | bwd_inner: 1800.12 | bwd_allreduce: 10.20 | step: 21.87 14%|█▍ | 405/2865 [2:43:01<2:44:03, 4.00s/it] {'loss': 0.8231, 'learning_rate': 3.871355178430062e-05, 'epoch': 0.42} 14%|█▍ | 405/2865 [2:43:01<2:44:03, 4.00s/it][INFO|trainer.py:3242] 2026-05-14 23:03:43,714 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 23:03:43,714 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 23:03:43,715 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-405 [INFO|configuration_utils.py:473] 2026-05-14 23:08:51,345 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-405/config.json [INFO|configuration_utils.py:594] 2026-05-14 23:08:51,356 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-405/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 23:08:57,272 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-405/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 23:08:57,282 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-405/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 23:08:57,289 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-405/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 23:08:57,291 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-405/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 23:09:06,367 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-330] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 303710.16 | bwd_microstep: 221.41 | bwd_inner_microstep: 221.18 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 160.85 | bwd_microstep: 219.00 | bwd_inner_microstep: 218.65 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 160.11 | bwd_microstep: 216.64 | bwd_inner_microstep: 216.45 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.54 | bwd_microstep: 215.54 | bwd_inner_microstep: 215.45 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 159.82 | bwd_microstep: 220.49 | bwd_inner_microstep: 220.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.44 | bwd_microstep: 217.01 | bwd_inner_microstep: 216.83 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 146.53 | bwd_microstep: 192.80 | bwd_inner_microstep: 192.65 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 162.40 | bwd_microstep: 249.66 | bwd_inner_microstep: 239.29 | bwd_allreduce_microstep: 10.16 | step_microstep: 21.28 [Rank 0] time (ms) | fwd: 304809.25 | bwd: 1752.67 | bwd_inner: 1741.22 | bwd_allreduce: 10.68 | step: 21.97 14%|█▍ | 406/2865 [2:48:27<68:47:54, 100.72s/it] {'loss': 1.1746, 'learning_rate': 3.870556192090299e-05, 'epoch': 0.42} 14%|█▍ | 406/2865 [2:48:27<68:47:54, 100.72s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.31 | bwd_microstep: 210.07 | bwd_inner_microstep: 209.88 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.74 | bwd_microstep: 215.82 | bwd_inner_microstep: 215.64 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.97 | bwd_microstep: 216.08 | bwd_inner_microstep: 215.94 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 159.99 | bwd_microstep: 213.51 | bwd_inner_microstep: 213.43 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.42 | bwd_microstep: 213.49 | bwd_inner_microstep: 213.37 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 161.23 | bwd_microstep: 214.57 | bwd_inner_microstep: 214.38 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 150.01 | bwd_microstep: 192.58 | bwd_inner_microstep: 192.17 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.95 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 147.44 | bwd_microstep: 220.32 | bwd_inner_microstep: 210.43 | bwd_allreduce_microstep: 9.72 | step_microstep: 21.68 [Rank 0] time (ms) | fwd: 1262.00 | bwd: 1696.58 | bwd_inner: 1685.53 | bwd_allreduce: 10.28 | step: 22.41 14%|█▍ | 407/2865 [2:48:30<48:46:40, 71.44s/it] {'loss': 0.8782, 'learning_rate': 3.869754815225124e-05, 'epoch': 0.43} 14%|█▍ | 407/2865 [2:48:30<48:46:40, 71.44s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 164.22 | bwd_microstep: 237.96 | bwd_inner_microstep: 237.39 | bwd_allreduce_microstep: 0.25 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 168.25 | bwd_microstep: 240.82 | bwd_inner_microstep: 240.33 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 165.58 | bwd_microstep: 240.61 | bwd_inner_microstep: 240.48 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 164.07 | bwd_microstep: 249.84 | bwd_inner_microstep: 249.35 | bwd_allreduce_microstep: 0.23 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 164.92 | bwd_microstep: 237.53 | bwd_inner_microstep: 237.11 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 165.80 | bwd_microstep: 239.76 | bwd_inner_microstep: 239.63 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 163.29 | bwd_microstep: 234.48 | bwd_inner_microstep: 234.35 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.94 | bwd_microstep: 244.77 | bwd_inner_microstep: 233.42 | bwd_allreduce_microstep: 11.25 | step_microstep: 21.14 [Rank 0] time (ms) | fwd: 1318.93 | bwd: 1926.01 | bwd_inner: 1912.47 | bwd_allreduce: 12.34 | step: 22.24 14%|█▍ | 408/2865 [2:48:33<34:49:53, 51.04s/it] {'loss': 0.7973, 'learning_rate': 3.868951048858678e-05, 'epoch': 0.43} 14%|█▍ | 408/2865 [2:48:33<34:49:53, 51.04s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 161.67 | bwd_microstep: 217.83 | bwd_inner_microstep: 217.59 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 162.20 | bwd_microstep: 217.00 | bwd_inner_microstep: 216.90 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 161.26 | bwd_microstep: 215.58 | bwd_inner_microstep: 215.40 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.71 | bwd_microstep: 211.69 | bwd_inner_microstep: 211.53 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.00 | bwd_microstep: 214.72 | bwd_inner_microstep: 214.49 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.66 | bwd_microstep: 217.38 | bwd_inner_microstep: 217.22 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 146.93 | bwd_microstep: 192.72 | bwd_inner_microstep: 192.39 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.83 | bwd_microstep: 254.02 | bwd_inner_microstep: 244.43 | bwd_allreduce_microstep: 9.51 | step_microstep: 21.08 [Rank 0] time (ms) | fwd: 1276.16 | bwd: 1741.06 | bwd_inner: 1730.24 | bwd_allreduce: 10.10 | step: 21.71 14%|█▍ | 409/2865 [2:48:37<25:01:20, 36.68s/it] {'loss': 0.9944, 'learning_rate': 3.868144894018155e-05, 'epoch': 0.43} 14%|█▍ | 409/2865 [2:48:37<25:01:20, 36.68s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 162.05 | bwd_microstep: 215.16 | bwd_inner_microstep: 214.95 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 161.43 | bwd_microstep: 218.97 | bwd_inner_microstep: 218.82 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 161.30 | bwd_microstep: 218.30 | bwd_inner_microstep: 217.90 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 177.13 | bwd_microstep: 212.95 | bwd_inner_microstep: 212.51 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.24 | bwd_microstep: 215.70 | bwd_inner_microstep: 215.33 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.73 | bwd_microstep: 215.69 | bwd_inner_microstep: 215.34 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.22 | bwd_microstep: 213.77 | bwd_inner_microstep: 213.46 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.27 | bwd_microstep: 248.60 | bwd_inner_microstep: 238.83 | bwd_allreduce_microstep: 9.69 | step_microstep: 21.04 [Rank 0] time (ms) | fwd: 1305.23 | bwd: 1759.33 | bwd_inner: 1747.58 | bwd_allreduce: 10.63 | step: 22.00 14%|█▍ | 410/2865 [2:48:40<18:10:19, 26.65s/it] {'loss': 0.8298, 'learning_rate': 3.8673363517338004e-05, 'epoch': 0.43} 14%|█▍ | 410/2865 [2:48:40<18:10:19, 26.65s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 160.60 | bwd_microstep: 215.30 | bwd_inner_microstep: 215.19 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 159.90 | bwd_microstep: 220.04 | bwd_inner_microstep: 219.65 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 162.44 | bwd_microstep: 219.03 | bwd_inner_microstep: 218.81 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.74 | bwd_microstep: 215.51 | bwd_inner_microstep: 215.37 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.68 | bwd_microstep: 214.54 | bwd_inner_microstep: 214.16 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 161.64 | bwd_microstep: 219.07 | bwd_inner_microstep: 218.96 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 161.45 | bwd_microstep: 225.91 | bwd_inner_microstep: 225.81 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.01 | bwd_microstep: 253.09 | bwd_inner_microstep: 242.69 | bwd_allreduce_microstep: 10.30 | step_microstep: 21.25 [Rank 0] time (ms) | fwd: 1289.34 | bwd: 1782.62 | bwd_inner: 1770.91 | bwd_allreduce: 10.92 | step: 21.99 14%|█▍ | 411/2865 [2:48:43<13:22:41, 19.63s/it] {'loss': 0.9659, 'learning_rate': 3.866525423038912e-05, 'epoch': 0.43} 14%|█▍ | 411/2865 [2:48:43<13:22:41, 19.63s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 943 [Rank 0] time (ms) | fwd_microstep: 162.91 | bwd_microstep: 216.10 | bwd_inner_microstep: 215.86 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 938 [Rank 0] time (ms) | fwd_microstep: 165.09 | bwd_microstep: 216.50 | bwd_inner_microstep: 216.39 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 162.14 | bwd_microstep: 215.20 | bwd_inner_microstep: 215.07 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 173.89 | bwd_microstep: 221.22 | bwd_inner_microstep: 220.84 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 162.40 | bwd_microstep: 222.33 | bwd_inner_microstep: 221.86 | bwd_allreduce_microstep: 0.22 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 164.68 | bwd_microstep: 221.38 | bwd_inner_microstep: 221.17 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 162.72 | bwd_microstep: 220.53 | bwd_inner_microstep: 220.20 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 169.57 | bwd_microstep: 253.94 | bwd_inner_microstep: 244.15 | bwd_allreduce_microstep: 9.68 | step_microstep: 21.16 [Rank 0] time (ms) | fwd: 1323.26 | bwd: 1787.36 | bwd_inner: 1775.92 | bwd_allreduce: 10.53 | step: 22.04 14%|█▍ | 412/2865 [2:48:46<10:02:10, 14.73s/it] {'loss': 0.9362, 'learning_rate': 3.8657121089698374e-05, 'epoch': 0.43} 14%|█▍ | 412/2865 [2:48:46<10:02:10, 14.73s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 164.63 | bwd_microstep: 219.68 | bwd_inner_microstep: 219.28 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 162.82 | bwd_microstep: 226.31 | bwd_inner_microstep: 226.00 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 161.67 | bwd_microstep: 221.84 | bwd_inner_microstep: 221.72 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 161.25 | bwd_microstep: 217.74 | bwd_inner_microstep: 217.55 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 162.57 | bwd_microstep: 217.69 | bwd_inner_microstep: 217.51 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 164.54 | bwd_microstep: 221.95 | bwd_inner_microstep: 221.66 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 163.07 | bwd_microstep: 223.36 | bwd_inner_microstep: 222.96 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 148.95 | bwd_microstep: 227.65 | bwd_inner_microstep: 216.19 | bwd_allreduce_microstep: 11.36 | step_microstep: 21.26 [Rank 0] time (ms) | fwd: 1289.40 | bwd: 1776.37 | bwd_inner: 1763.25 | bwd_allreduce: 12.17 | step: 21.97 14%|█▍ | 413/2865 [2:48:50<7:41:01, 11.28s/it] {'loss': 0.844, 'learning_rate': 3.8648964105659713e-05, 'epoch': 0.43} 14%|█▍ | 413/2865 [2:48:50<7:41:01, 11.28s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.77 | bwd_microstep: 220.74 | bwd_inner_microstep: 220.63 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.78 | bwd_microstep: 221.41 | bwd_inner_microstep: 221.00 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 161.61 | bwd_microstep: 224.57 | bwd_inner_microstep: 224.42 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 162.10 | bwd_microstep: 219.18 | bwd_inner_microstep: 219.08 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.85 | bwd_microstep: 226.65 | bwd_inner_microstep: 226.55 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.32 | bwd_microstep: 219.39 | bwd_inner_microstep: 219.29 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 161.55 | bwd_microstep: 219.33 | bwd_inner_microstep: 219.22 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.16 | bwd_microstep: 262.63 | bwd_inner_microstep: 252.45 | bwd_allreduce_microstep: 10.09 | step_microstep: 21.35 [Rank 0] time (ms) | fwd: 1294.05 | bwd: 1814.02 | bwd_inner: 1802.76 | bwd_allreduce: 10.64 | step: 22.01 14%|█▍ | 414/2865 [2:48:53<6:02:37, 8.88s/it] {'loss': 0.9089, 'learning_rate': 3.864078328869758e-05, 'epoch': 0.43} 14%|█▍ | 414/2865 [2:48:53<6:02:37, 8.88s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 162.13 | bwd_microstep: 214.44 | bwd_inner_microstep: 214.19 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.84 | bwd_microstep: 218.63 | bwd_inner_microstep: 218.43 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.17 | bwd_microstep: 220.57 | bwd_inner_microstep: 220.43 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 163.65 | bwd_microstep: 216.33 | bwd_inner_microstep: 216.20 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 163.59 | bwd_microstep: 214.83 | bwd_inner_microstep: 214.64 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 161.21 | bwd_microstep: 218.33 | bwd_inner_microstep: 218.21 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.64 | bwd_microstep: 221.48 | bwd_inner_microstep: 221.35 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.05 | bwd_microstep: 250.63 | bwd_inner_microstep: 239.03 | bwd_allreduce_microstep: 11.47 | step_microstep: 21.07 [Rank 0] time (ms) | fwd: 1297.17 | bwd: 1775.32 | bwd_inner: 1762.76 | bwd_allreduce: 11.91 | step: 21.67 14%|█▍ | 415/2865 [2:48:56<4:53:33, 7.19s/it] {'loss': 0.8935, 'learning_rate': 3.863257864926684e-05, 'epoch': 0.43} 14%|█▍ | 415/2865 [2:48:56<4:53:33, 7.19s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 162.54 | bwd_microstep: 220.85 | bwd_inner_microstep: 220.64 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 164.90 | bwd_microstep: 218.28 | bwd_inner_microstep: 218.17 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 161.55 | bwd_microstep: 220.22 | bwd_inner_microstep: 220.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 942 [Rank 0] time (ms) | fwd_microstep: 163.50 | bwd_microstep: 222.81 | bwd_inner_microstep: 222.66 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 163.58 | bwd_microstep: 225.63 | bwd_inner_microstep: 225.50 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 146.56 | bwd_microstep: 191.97 | bwd_inner_microstep: 191.83 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 162.45 | bwd_microstep: 215.45 | bwd_inner_microstep: 215.37 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 163.80 | bwd_microstep: 247.33 | bwd_inner_microstep: 237.66 | bwd_allreduce_microstep: 9.59 | step_microstep: 21.10 [Rank 0] time (ms) | fwd: 1288.75 | bwd: 1762.63 | bwd_inner: 1752.07 | bwd_allreduce: 10.05 | step: 21.68 15%|█▍ | 416/2865 [2:48:59<4:04:48, 6.00s/it] {'loss': 1.1008, 'learning_rate': 3.862435019785284e-05, 'epoch': 0.44} 15%|█▍ | 416/2865 [2:48:59<4:04:48, 6.00s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 941 [Rank 0] time (ms) | fwd_microstep: 164.11 | bwd_microstep: 214.97 | bwd_inner_microstep: 214.83 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 162.01 | bwd_microstep: 218.27 | bwd_inner_microstep: 217.90 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.24 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 161.19 | bwd_microstep: 220.15 | bwd_inner_microstep: 219.77 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 162.33 | bwd_microstep: 273.08 | bwd_inner_microstep: 272.74 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.66 | bwd_microstep: 216.20 | bwd_inner_microstep: 215.89 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 162.57 | bwd_microstep: 214.56 | bwd_inner_microstep: 214.27 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 163.97 | bwd_microstep: 214.72 | bwd_inner_microstep: 214.49 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.28 | bwd_microstep: 247.12 | bwd_inner_microstep: 237.32 | bwd_allreduce_microstep: 9.69 | step_microstep: 20.97 [Rank 0] time (ms) | fwd: 1299.99 | bwd: 1819.19 | bwd_inner: 1807.74 | bwd_allreduce: 10.44 | step: 21.80 15%|█▍ | 417/2865 [2:49:03<3:31:27, 5.18s/it] {'loss': 1.0733, 'learning_rate': 3.861609794497134e-05, 'epoch': 0.44} 15%|█▍ | 417/2865 [2:49:03<3:31:27, 5.18s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 164.92 | bwd_microstep: 225.59 | bwd_inner_microstep: 225.30 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 162.48 | bwd_microstep: 218.63 | bwd_inner_microstep: 218.33 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 163.07 | bwd_microstep: 218.51 | bwd_inner_microstep: 218.21 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 162.19 | bwd_microstep: 222.76 | bwd_inner_microstep: 222.63 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 163.95 | bwd_microstep: 219.18 | bwd_inner_microstep: 218.88 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 165.88 | bwd_microstep: 252.78 | bwd_inner_microstep: 252.41 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 160.22 | bwd_microstep: 236.33 | bwd_inner_microstep: 236.20 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.06 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 172.67 | bwd_microstep: 284.15 | bwd_inner_microstep: 270.13 | bwd_allreduce_microstep: 13.89 | step_microstep: 23.34 [Rank 0] time (ms) | fwd: 1315.26 | bwd: 1878.09 | bwd_inner: 1862.51 | bwd_allreduce: 14.70 | step: 24.17 15%|█▍ | 418/2865 [2:49:06<3:09:08, 4.64s/it] {'loss': 0.8575, 'learning_rate': 3.860782190116851e-05, 'epoch': 0.44} 15%|█▍ | 418/2865 [2:49:06<3:09:08, 4.64s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 167.94 | bwd_microstep: 251.29 | bwd_inner_microstep: 251.15 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 166.30 | bwd_microstep: 225.86 | bwd_inner_microstep: 225.50 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.52 | bwd_microstep: 223.86 | bwd_inner_microstep: 223.66 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.27 | bwd_microstep: 222.42 | bwd_inner_microstep: 222.06 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 161.24 | bwd_microstep: 220.54 | bwd_inner_microstep: 220.10 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.59 | bwd_microstep: 222.47 | bwd_inner_microstep: 222.09 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 161.30 | bwd_microstep: 219.30 | bwd_inner_microstep: 218.99 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.53 | bwd_microstep: 249.36 | bwd_inner_microstep: 238.15 | bwd_allreduce_microstep: 11.03 | step_microstep: 21.26 [Rank 0] time (ms) | fwd: 1306.57 | bwd: 1835.28 | bwd_inner: 1822.22 | bwd_allreduce: 11.88 | step: 22.28 15%|█▍ | 419/2865 [2:49:09<2:52:54, 4.24s/it] {'loss': 0.9333, 'learning_rate': 3.859952207702093e-05, 'epoch': 0.44} 15%|█▍ | 419/2865 [2:49:09<2:52:54, 4.24s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 161.23 | bwd_microstep: 203.30 | bwd_inner_microstep: 203.00 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 162.78 | bwd_microstep: 222.59 | bwd_inner_microstep: 222.51 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 159.80 | bwd_microstep: 217.22 | bwd_inner_microstep: 216.91 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 161.31 | bwd_microstep: 222.40 | bwd_inner_microstep: 222.16 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 161.08 | bwd_microstep: 217.73 | bwd_inner_microstep: 217.60 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.97 | bwd_microstep: 214.79 | bwd_inner_microstep: 214.46 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.49 | bwd_microstep: 216.82 | bwd_inner_microstep: 216.50 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.02 | bwd_microstep: 252.00 | bwd_inner_microstep: 242.33 | bwd_allreduce_microstep: 9.53 | step_microstep: 20.87 [Rank 0] time (ms) | fwd: 1291.57 | bwd: 1766.93 | bwd_inner: 1755.90 | bwd_allreduce: 10.26 | step: 21.42 15%|█▍ | 420/2865 [2:49:13<2:40:22, 3.94s/it] {'loss': 1.0578, 'learning_rate': 3.859119848313559e-05, 'epoch': 0.44} 15%|█▍ | 420/2865 [2:49:13<2:40:22, 3.94s/it][INFO|trainer.py:3242] 2026-05-14 23:09:55,778 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 23:09:55,778 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 23:09:55,778 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-420 [INFO|configuration_utils.py:473] 2026-05-14 23:15:02,824 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-420/config.json [INFO|configuration_utils.py:594] 2026-05-14 23:15:02,832 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-420/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 23:15:09,112 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-420/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 23:15:09,126 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-420/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 23:15:09,134 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-420/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 23:15:09,138 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-420/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 23:15:20,190 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-345] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 303932.30 | bwd_microstep: 264.56 | bwd_inner_microstep: 264.08 | bwd_allreduce_microstep: 0.29 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 182.85 | bwd_microstep: 261.83 | bwd_inner_microstep: 261.62 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 187.10 | bwd_microstep: 272.26 | bwd_inner_microstep: 271.84 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 180.43 | bwd_microstep: 217.00 | bwd_inner_microstep: 216.67 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 162.33 | bwd_microstep: 215.47 | bwd_inner_microstep: 215.25 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 162.49 | bwd_microstep: 216.94 | bwd_inner_microstep: 216.69 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 163.53 | bwd_microstep: 217.00 | bwd_inner_microstep: 216.84 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.55 | bwd_microstep: 245.41 | bwd_inner_microstep: 235.37 | bwd_allreduce_microstep: 9.97 | step_microstep: 21.67 [Rank 0] time (ms) | fwd: 305121.65 | bwd: 1910.62 | bwd_inner: 1898.75 | bwd_allreduce: 10.95 | step: 22.56 15%|█▍ | 421/2865 [2:54:41<68:43:53, 101.24s/it] {'loss': 0.9433, 'learning_rate': 3.858285113014983e-05, 'epoch': 0.44} 15%|█▍ | 421/2865 [2:54:41<68:43:53, 101.24s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 162.01 | bwd_microstep: 216.40 | bwd_inner_microstep: 216.27 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.84 | bwd_microstep: 215.48 | bwd_inner_microstep: 215.35 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.77 | bwd_microstep: 218.37 | bwd_inner_microstep: 218.25 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.22 | bwd_microstep: 215.60 | bwd_inner_microstep: 215.34 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 162.56 | bwd_microstep: 223.43 | bwd_inner_microstep: 223.17 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 161.84 | bwd_microstep: 218.09 | bwd_inner_microstep: 217.92 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 171.81 | bwd_microstep: 224.84 | bwd_inner_microstep: 224.70 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.86 [Rank 0] time (ms) | fwd_microstep: 162.57 | bwd_microstep: 260.64 | bwd_inner_microstep: 250.70 | bwd_allreduce_microstep: 9.76 | step_microstep: 21.67 [Rank 0] time (ms) | fwd: 1302.49 | bwd: 1792.95 | bwd_inner: 1782.03 | bwd_allreduce: 10.20 | step: 22.34 15%|█▍ | 422/2865 [2:54:44<48:45:21, 71.85s/it] {'loss': 0.7995, 'learning_rate': 3.8574480028731364e-05, 'epoch': 0.44} 15%|█▍ | 422/2865 [2:54:44<48:45:21, 71.85s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 162.46 | bwd_microstep: 222.53 | bwd_inner_microstep: 222.11 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 161.70 | bwd_microstep: 216.11 | bwd_inner_microstep: 215.79 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 165.41 | bwd_microstep: 225.14 | bwd_inner_microstep: 224.88 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 163.48 | bwd_microstep: 225.12 | bwd_inner_microstep: 224.98 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 164.05 | bwd_microstep: 223.20 | bwd_inner_microstep: 222.93 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 165.23 | bwd_microstep: 219.55 | bwd_inner_microstep: 219.28 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 162.21 | bwd_microstep: 225.92 | bwd_inner_microstep: 225.34 | bwd_allreduce_microstep: 0.23 | step_microstep: 0.31 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 160.91 | bwd_microstep: 259.11 | bwd_inner_microstep: 249.23 | bwd_allreduce_microstep: 9.74 | step_microstep: 21.73 [Rank 0] time (ms) | fwd: 1305.32 | bwd: 1816.88 | bwd_inner: 1805.04 | bwd_allreduce: 10.66 | step: 22.82 15%|█▍ | 423/2865 [2:54:47<34:47:15, 51.28s/it] {'loss': 1.1884, 'learning_rate': 3.856608518957827e-05, 'epoch': 0.44} 15%|█▍ | 423/2865 [2:54:47<34:47:15, 51.28s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 161.58 | bwd_microstep: 231.07 | bwd_inner_microstep: 230.79 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.52 | bwd_microstep: 216.85 | bwd_inner_microstep: 216.47 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 162.04 | bwd_microstep: 223.40 | bwd_inner_microstep: 223.27 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 168.38 | bwd_microstep: 217.37 | bwd_inner_microstep: 217.08 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.75 | bwd_microstep: 217.51 | bwd_inner_microstep: 217.37 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 175.87 | bwd_microstep: 217.53 | bwd_inner_microstep: 217.35 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.40 | bwd_microstep: 216.92 | bwd_inner_microstep: 216.62 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 166.63 | bwd_microstep: 248.17 | bwd_inner_microstep: 237.15 | bwd_allreduce_microstep: 10.85 | step_microstep: 21.10 [Rank 0] time (ms) | fwd: 1317.04 | bwd: 1788.93 | bwd_inner: 1776.55 | bwd_allreduce: 11.47 | step: 21.73 15%|█▍ | 424/2865 [2:54:51<25:00:29, 36.88s/it] {'loss': 0.8271, 'learning_rate': 3.8557666623418925e-05, 'epoch': 0.44} 15%|█▍ | 424/2865 [2:54:51<25:00:29, 36.88s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 938 [Rank 0] time (ms) | fwd_microstep: 162.12 | bwd_microstep: 218.73 | bwd_inner_microstep: 218.50 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 160.36 | bwd_microstep: 224.89 | bwd_inner_microstep: 224.54 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 162.32 | bwd_microstep: 216.86 | bwd_inner_microstep: 216.74 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.10 | bwd_microstep: 198.39 | bwd_inner_microstep: 198.26 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.20 | bwd_microstep: 190.18 | bwd_inner_microstep: 189.81 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 162.17 | bwd_microstep: 191.24 | bwd_inner_microstep: 191.11 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 160.78 | bwd_microstep: 190.04 | bwd_inner_microstep: 189.83 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 166.69 | bwd_microstep: 218.51 | bwd_inner_microstep: 208.01 | bwd_allreduce_microstep: 10.35 | step_microstep: 21.22 [Rank 0] time (ms) | fwd: 1296.63 | bwd: 1648.97 | bwd_inner: 1637.20 | bwd_allreduce: 10.94 | step: 21.94 15%|█▍ | 425/2865 [2:54:54<18:07:47, 26.75s/it] {'loss': 0.6884, 'learning_rate': 3.854922434101207e-05, 'epoch': 0.44} 15%|█▍ | 425/2865 [2:54:54<18:07:47, 26.75s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.63 | bwd_microstep: 187.49 | bwd_inner_microstep: 187.27 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.41 | bwd_microstep: 187.60 | bwd_inner_microstep: 187.54 | bwd_allreduce_microstep: 0.02 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.40 | bwd_microstep: 188.93 | bwd_inner_microstep: 188.60 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 162.42 | bwd_microstep: 189.44 | bwd_inner_microstep: 189.23 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.43 | bwd_microstep: 190.26 | bwd_inner_microstep: 190.16 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.01 | bwd_microstep: 190.33 | bwd_inner_microstep: 190.23 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.73 | bwd_microstep: 189.08 | bwd_inner_microstep: 188.86 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 150.50 | bwd_microstep: 204.97 | bwd_inner_microstep: 194.59 | bwd_allreduce_microstep: 10.28 | step_microstep: 20.96 [Rank 0] time (ms) | fwd: 1276.41 | bwd: 1528.19 | bwd_inner: 1516.77 | bwd_allreduce: 10.75 | step: 21.47 15%|█▍ | 426/2865 [2:54:57<13:17:19, 19.61s/it] {'loss': 0.9837, 'learning_rate': 3.854075835314674e-05, 'epoch': 0.45} 15%|█▍ | 426/2865 [2:54:57<13:17:19, 19.61s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 178.19 | bwd_microstep: 198.84 | bwd_inner_microstep: 198.60 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 204.19 | bwd_microstep: 199.85 | bwd_inner_microstep: 199.73 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 161.86 | bwd_microstep: 206.88 | bwd_inner_microstep: 206.75 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 173.66 | bwd_microstep: 195.35 | bwd_inner_microstep: 195.21 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.26 | bwd_microstep: 190.16 | bwd_inner_microstep: 189.95 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 160.91 | bwd_microstep: 189.56 | bwd_inner_microstep: 189.24 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.61 | bwd_microstep: 221.67 | bwd_inner_microstep: 221.29 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.33 | bwd_microstep: 251.59 | bwd_inner_microstep: 239.91 | bwd_allreduce_microstep: 11.60 | step_microstep: 21.11 [Rank 0] time (ms) | fwd: 1361.91 | bwd: 1654.01 | bwd_inner: 1640.97 | bwd_allreduce: 12.32 | step: 21.74 15%|█▍ | 427/2865 [2:55:00<9:56:38, 14.68s/it] {'loss': 0.7566, 'learning_rate': 3.853226867064224e-05, 'epoch': 0.45} 15%|█▍ | 427/2865 [2:55:00<9:56:38, 14.68s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 161.35 | bwd_microstep: 186.72 | bwd_inner_microstep: 186.61 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 161.50 | bwd_microstep: 201.57 | bwd_inner_microstep: 201.45 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 162.45 | bwd_microstep: 202.26 | bwd_inner_microstep: 202.15 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 175.28 | bwd_microstep: 201.47 | bwd_inner_microstep: 201.33 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 159.95 | bwd_microstep: 216.24 | bwd_inner_microstep: 215.88 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.05 | bwd_microstep: 215.62 | bwd_inner_microstep: 215.50 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 159.04 | bwd_microstep: 219.54 | bwd_inner_microstep: 219.15 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 177.11 | bwd_microstep: 251.58 | bwd_inner_microstep: 239.22 | bwd_allreduce_microstep: 12.19 | step_microstep: 21.49 [Rank 0] time (ms) | fwd: 1316.61 | bwd: 1695.13 | bwd_inner: 1681.58 | bwd_allreduce: 12.79 | step: 22.25 15%|█▍ | 428/2865 [2:55:03<7:36:09, 11.23s/it] {'loss': 1.0066, 'learning_rate': 3.85237553043482e-05, 'epoch': 0.45} 15%|█▍ | 428/2865 [2:55:03<7:36:09, 11.23s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 159.79 | bwd_microstep: 214.26 | bwd_inner_microstep: 213.90 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.15 | bwd_microstep: 215.49 | bwd_inner_microstep: 215.36 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 160.34 | bwd_microstep: 216.66 | bwd_inner_microstep: 216.29 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.08 | bwd_microstep: 212.49 | bwd_inner_microstep: 212.36 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.04 | bwd_microstep: 215.00 | bwd_inner_microstep: 214.86 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 159.77 | bwd_microstep: 215.08 | bwd_inner_microstep: 214.71 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 161.49 | bwd_microstep: 213.92 | bwd_inner_microstep: 213.78 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.56 | bwd_microstep: 248.01 | bwd_inner_microstep: 238.26 | bwd_allreduce_microstep: 9.59 | step_microstep: 21.12 [Rank 0] time (ms) | fwd: 1282.11 | bwd: 1751.06 | bwd_inner: 1739.89 | bwd_allreduce: 10.25 | step: 21.98 15%|█▍ | 429/2865 [2:55:06<5:58:05, 8.82s/it] {'loss': 0.931, 'learning_rate': 3.851521826514448e-05, 'epoch': 0.45} 15%|█▍ | 429/2865 [2:55:06<5:58:05, 8.82s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 160.87 | bwd_microstep: 213.00 | bwd_inner_microstep: 212.87 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 160.56 | bwd_microstep: 214.78 | bwd_inner_microstep: 214.65 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.49 | bwd_microstep: 214.08 | bwd_inner_microstep: 213.95 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 159.58 | bwd_microstep: 215.33 | bwd_inner_microstep: 215.21 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.00 | bwd_microstep: 211.72 | bwd_inner_microstep: 211.58 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 159.91 | bwd_microstep: 216.03 | bwd_inner_microstep: 215.89 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.25 | bwd_microstep: 215.25 | bwd_inner_microstep: 214.87 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.57 | bwd_microstep: 254.16 | bwd_inner_microstep: 243.55 | bwd_allreduce_microstep: 10.53 | step_microstep: 21.48 [Rank 0] time (ms) | fwd: 1282.14 | bwd: 1754.45 | bwd_inner: 1742.79 | bwd_allreduce: 11.06 | step: 22.06 15%|█▌ | 430/2865 [2:55:10<4:49:25, 7.13s/it] {'loss': 0.6008, 'learning_rate': 3.850665756394123e-05, 'epoch': 0.45} 15%|█▌ | 430/2865 [2:55:10<4:49:25, 7.13s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 163.98 | bwd_microstep: 241.07 | bwd_inner_microstep: 240.87 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 165.94 | bwd_microstep: 258.16 | bwd_inner_microstep: 258.02 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 160.41 | bwd_microstep: 212.93 | bwd_inner_microstep: 212.60 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.22 | bwd_microstep: 212.36 | bwd_inner_microstep: 212.18 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.06 | bwd_microstep: 188.79 | bwd_inner_microstep: 188.57 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.56 | bwd_microstep: 190.31 | bwd_inner_microstep: 189.95 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 146.39 | bwd_microstep: 176.56 | bwd_inner_microstep: 176.25 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 159.79 | bwd_microstep: 239.34 | bwd_inner_microstep: 229.42 | bwd_allreduce_microstep: 9.83 | step_microstep: 21.12 [Rank 0] time (ms) | fwd: 1277.25 | bwd: 1719.65 | bwd_inner: 1708.31 | bwd_allreduce: 10.49 | step: 21.75 15%|█▌ | 431/2865 [2:55:13<4:01:01, 5.94s/it] {'loss': 0.885, 'learning_rate': 3.8498073211678795e-05, 'epoch': 0.45} 15%|█▌ | 431/2865 [2:55:13<4:01:01, 5.94s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 161.76 | bwd_microstep: 188.78 | bwd_inner_microstep: 188.55 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 160.82 | bwd_microstep: 191.04 | bwd_inner_microstep: 190.81 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.19 | bwd_microstep: 188.94 | bwd_inner_microstep: 188.58 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.59 | bwd_microstep: 192.61 | bwd_inner_microstep: 192.36 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.68 | bwd_microstep: 186.67 | bwd_inner_microstep: 186.54 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.12 | bwd_microstep: 189.29 | bwd_inner_microstep: 188.93 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 159.90 | bwd_microstep: 188.73 | bwd_inner_microstep: 188.62 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.31 | bwd_microstep: 215.70 | bwd_inner_microstep: 205.88 | bwd_allreduce_microstep: 9.74 | step_microstep: 21.16 [Rank 0] time (ms) | fwd: 1285.25 | bwd: 1541.91 | bwd_inner: 1530.60 | bwd_allreduce: 10.44 | step: 22.00 15%|█▌ | 432/2865 [2:55:16<3:25:02, 5.06s/it] {'loss': 0.9345, 'learning_rate': 3.8489465219327786e-05, 'epoch': 0.45} 15%|█▌ | 432/2865 [2:55:16<3:25:02, 5.06s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 161.00 | bwd_microstep: 202.90 | bwd_inner_microstep: 202.53 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 161.53 | bwd_microstep: 429.12 | bwd_inner_microstep: 428.76 | bwd_allreduce_microstep: 0.20 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 160.27 | bwd_microstep: 189.62 | bwd_inner_microstep: 189.51 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.43 | bwd_microstep: 188.58 | bwd_inner_microstep: 188.47 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 161.41 | bwd_microstep: 191.75 | bwd_inner_microstep: 191.60 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.33 | bwd_microstep: 194.09 | bwd_inner_microstep: 193.97 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 172.22 | bwd_microstep: 191.65 | bwd_inner_microstep: 191.36 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.30 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 162.60 | bwd_microstep: 237.86 | bwd_inner_microstep: 220.92 | bwd_allreduce_microstep: 16.79 | step_microstep: 28.91 [Rank 0] time (ms) | fwd: 1299.68 | bwd: 1825.71 | bwd_inner: 1807.46 | bwd_allreduce: 17.47 | step: 29.54 15%|█▌ | 433/2865 [2:55:19<3:03:34, 4.53s/it] {'loss': 0.794, 'learning_rate': 3.8480833597888997e-05, 'epoch': 0.45} 15%|█▌ | 433/2865 [2:55:19<3:03:34, 4.53s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 164.60 | bwd_microstep: 192.67 | bwd_inner_microstep: 192.55 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 168.07 | bwd_microstep: 193.58 | bwd_inner_microstep: 193.45 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 164.11 | bwd_microstep: 203.01 | bwd_inner_microstep: 202.88 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.41 | bwd_microstep: 193.16 | bwd_inner_microstep: 192.89 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.10 | bwd_microstep: 194.54 | bwd_inner_microstep: 194.42 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 161.07 | bwd_microstep: 190.35 | bwd_inner_microstep: 190.23 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 165.83 | bwd_microstep: 189.86 | bwd_inner_microstep: 189.48 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 159.56 | bwd_microstep: 247.60 | bwd_inner_microstep: 237.82 | bwd_allreduce_microstep: 9.69 | step_microstep: 21.14 [Rank 0] time (ms) | fwd: 1308.65 | bwd: 1604.89 | bwd_inner: 1593.98 | bwd_allreduce: 10.21 | step: 21.77 15%|█▌ | 434/2865 [2:55:22<2:45:51, 4.09s/it] {'loss': 0.6602, 'learning_rate': 3.847217835839344e-05, 'epoch': 0.45} 15%|█▌ | 434/2865 [2:55:22<2:45:51, 4.09s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 161.00 | bwd_microstep: 214.79 | bwd_inner_microstep: 214.25 | bwd_allreduce_microstep: 0.22 | step_microstep: 0.27 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 165.38 | bwd_microstep: 222.58 | bwd_inner_microstep: 222.19 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.60 | bwd_microstep: 227.11 | bwd_inner_microstep: 226.99 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 161.48 | bwd_microstep: 222.21 | bwd_inner_microstep: 221.96 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 163.53 | bwd_microstep: 308.27 | bwd_inner_microstep: 308.00 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 163.36 | bwd_microstep: 226.60 | bwd_inner_microstep: 226.31 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 163.75 | bwd_microstep: 222.97 | bwd_inner_microstep: 222.83 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.95 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 147.47 | bwd_microstep: 238.69 | bwd_inner_microstep: 227.50 | bwd_allreduce_microstep: 11.04 | step_microstep: 21.21 [Rank 0] time (ms) | fwd: 1287.44 | bwd: 1883.39 | bwd_inner: 1870.58 | bwd_allreduce: 11.77 | step: 22.06 15%|█▌ | 435/2865 [2:55:25<2:36:37, 3.87s/it] {'loss': 0.7165, 'learning_rate': 3.846349951190228e-05, 'epoch': 0.46} 15%|█▌ | 435/2865 [2:55:25<2:36:37, 3.87s/it][INFO|trainer.py:3242] 2026-05-14 23:16:08,590 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 23:16:08,591 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 23:16:08,591 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-435 [INFO|configuration_utils.py:473] 2026-05-14 23:21:15,172 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-435/config.json [INFO|configuration_utils.py:594] 2026-05-14 23:21:15,179 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-435/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 23:21:20,492 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-435/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 23:21:20,513 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-435/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 23:21:20,520 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-435/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 23:21:20,522 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-435/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 23:21:28,874 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-360] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 948 [Rank 0] time (ms) | fwd_microstep: 303596.21 | bwd_microstep: 226.86 | bwd_inner_microstep: 226.47 | bwd_allreduce_microstep: 0.20 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 175.91 | bwd_microstep: 223.57 | bwd_inner_microstep: 223.45 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 164.39 | bwd_microstep: 223.14 | bwd_inner_microstep: 222.75 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.10 | bwd_microstep: 224.84 | bwd_inner_microstep: 224.48 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 162.03 | bwd_microstep: 222.32 | bwd_inner_microstep: 221.94 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 160.61 | bwd_microstep: 219.91 | bwd_inner_microstep: 219.56 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 163.28 | bwd_microstep: 221.44 | bwd_inner_microstep: 221.22 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.98 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.35 | bwd_microstep: 255.86 | bwd_inner_microstep: 243.39 | bwd_allreduce_microstep: 12.31 | step_microstep: 22.51 [Rank 0] time (ms) | fwd: 304732.42 | bwd: 1818.11 | bwd_inner: 1803.75 | bwd_allreduce: 13.25 | step: 23.39 15%|█▌ | 436/2865 [3:00:49<67:22:55, 99.87s/it] {'loss': 1.0008, 'learning_rate': 3.84547970695069e-05, 'epoch': 0.46} 15%|█▌ | 436/2865 [3:00:49<67:22:55, 99.87s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 160.67 | bwd_microstep: 217.71 | bwd_inner_microstep: 217.48 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 160.28 | bwd_microstep: 299.43 | bwd_inner_microstep: 299.24 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 169.41 | bwd_microstep: 255.39 | bwd_inner_microstep: 255.11 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 170.22 | bwd_microstep: 256.98 | bwd_inner_microstep: 256.83 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 168.37 | bwd_microstep: 254.42 | bwd_inner_microstep: 253.94 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.24 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 166.35 | bwd_microstep: 240.23 | bwd_inner_microstep: 240.03 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 164.61 | bwd_microstep: 228.74 | bwd_inner_microstep: 228.35 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 164.72 | bwd_microstep: 253.27 | bwd_inner_microstep: 241.92 | bwd_allreduce_microstep: 11.18 | step_microstep: 21.40 [Rank 0] time (ms) | fwd: 1324.48 | bwd: 2006.30 | bwd_inner: 1993.42 | bwd_allreduce: 11.90 | step: 22.18 15%|█▌ | 437/2865 [3:00:53<47:51:33, 70.96s/it] {'loss': 0.7078, 'learning_rate': 3.8446071042328796e-05, 'epoch': 0.46} 15%|█▌ | 437/2865 [3:00:53<47:51:33, 70.96s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 161.45 | bwd_microstep: 221.13 | bwd_inner_microstep: 220.98 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 161.59 | bwd_microstep: 223.16 | bwd_inner_microstep: 223.02 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 162.60 | bwd_microstep: 217.98 | bwd_inner_microstep: 217.60 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 164.25 | bwd_microstep: 220.27 | bwd_inner_microstep: 219.94 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 165.52 | bwd_microstep: 221.43 | bwd_inner_microstep: 221.08 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 164.47 | bwd_microstep: 218.33 | bwd_inner_microstep: 218.14 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.13 | bwd_microstep: 215.16 | bwd_inner_microstep: 215.04 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.88 | bwd_microstep: 255.75 | bwd_inner_microstep: 245.27 | bwd_allreduce_microstep: 10.30 | step_microstep: 21.44 [Rank 0] time (ms) | fwd: 1301.79 | bwd: 1793.35 | bwd_inner: 1781.44 | bwd_allreduce: 10.96 | step: 22.20 15%|█▌ | 438/2865 [3:00:56<34:08:48, 50.65s/it] {'loss': 0.7031, 'learning_rate': 3.8437321441519627e-05, 'epoch': 0.46} 15%|█▌ | 438/2865 [3:00:56<34:08:48, 50.65s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 165.89 | bwd_microstep: 215.66 | bwd_inner_microstep: 215.52 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 162.15 | bwd_microstep: 218.26 | bwd_inner_microstep: 217.81 | bwd_allreduce_microstep: 0.21 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.63 | bwd_microstep: 226.40 | bwd_inner_microstep: 226.04 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 163.08 | bwd_microstep: 216.63 | bwd_inner_microstep: 216.35 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 165.77 | bwd_microstep: 224.17 | bwd_inner_microstep: 223.86 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 162.66 | bwd_microstep: 224.46 | bwd_inner_microstep: 224.15 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.54 | bwd_microstep: 221.77 | bwd_inner_microstep: 221.56 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.43 | bwd_microstep: 260.15 | bwd_inner_microstep: 248.75 | bwd_allreduce_microstep: 11.22 | step_microstep: 21.36 [Rank 0] time (ms) | fwd: 1305.06 | bwd: 1807.65 | bwd_inner: 1794.54 | bwd_allreduce: 12.05 | step: 22.05 15%|█▌ | 439/2865 [3:00:59<24:33:13, 36.44s/it] {'loss': 0.8728, 'learning_rate': 3.8428548278261156e-05, 'epoch': 0.46} 15%|█▌ | 439/2865 [3:00:59<24:33:13, 36.44s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 162.61 | bwd_microstep: 219.15 | bwd_inner_microstep: 218.85 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.66 | bwd_microstep: 223.61 | bwd_inner_microstep: 223.39 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 150.65 | bwd_microstep: 201.36 | bwd_inner_microstep: 201.01 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 161.38 | bwd_microstep: 225.43 | bwd_inner_microstep: 225.03 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.01 | bwd_microstep: 221.14 | bwd_inner_microstep: 221.00 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 146.93 | bwd_microstep: 192.79 | bwd_inner_microstep: 192.56 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 167.22 | bwd_microstep: 223.30 | bwd_inner_microstep: 222.94 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 166.13 | bwd_microstep: 251.20 | bwd_inner_microstep: 241.15 | bwd_allreduce_microstep: 9.96 | step_microstep: 20.88 [Rank 0] time (ms) | fwd: 1278.48 | bwd: 1758.15 | bwd_inner: 1746.36 | bwd_allreduce: 10.78 | step: 21.72 15%|█▌ | 440/2865 [3:01:03<17:49:34, 26.46s/it] {'loss': 0.8652, 'learning_rate': 3.841975156376529e-05, 'epoch': 0.46} 15%|█▌ | 440/2865 [3:01:03<17:49:34, 26.46s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 162.84 | bwd_microstep: 218.25 | bwd_inner_microstep: 218.04 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 162.97 | bwd_microstep: 222.46 | bwd_inner_microstep: 222.05 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.71 | bwd_microstep: 222.20 | bwd_inner_microstep: 222.07 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 164.55 | bwd_microstep: 222.53 | bwd_inner_microstep: 222.14 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 166.11 | bwd_microstep: 224.19 | bwd_inner_microstep: 223.88 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 149.04 | bwd_microstep: 196.57 | bwd_inner_microstep: 195.97 | bwd_allreduce_microstep: 0.35 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 162.04 | bwd_microstep: 226.14 | bwd_inner_microstep: 226.01 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 909 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.28 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.34 | bwd_microstep: 257.37 | bwd_inner_microstep: 245.76 | bwd_allreduce_microstep: 11.44 | step_microstep: 22.87 [Rank 0] time (ms) | fwd: 1291.50 | bwd: 1789.86 | bwd_inner: 1776.38 | bwd_allreduce: 12.44 | step: 23.78 15%|█▌ | 441/2865 [3:01:06<13:07:41, 19.50s/it] {'loss': 1.0756, 'learning_rate': 3.8410931309273996e-05, 'epoch': 0.46} 15%|█▌ | 441/2865 [3:01:06<13:07:41, 19.50s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 162.58 | bwd_microstep: 217.46 | bwd_inner_microstep: 217.32 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 163.69 | bwd_microstep: 226.24 | bwd_inner_microstep: 225.89 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 163.81 | bwd_microstep: 221.02 | bwd_inner_microstep: 220.80 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 164.06 | bwd_microstep: 225.40 | bwd_inner_microstep: 225.10 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 163.29 | bwd_microstep: 226.98 | bwd_inner_microstep: 226.62 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 162.25 | bwd_microstep: 231.43 | bwd_inner_microstep: 231.14 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 162.38 | bwd_microstep: 229.94 | bwd_inner_microstep: 229.75 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.82 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 244.75 | bwd_microstep: 248.48 | bwd_inner_microstep: 238.72 | bwd_allreduce_microstep: 9.60 | step_microstep: 20.80 [Rank 0] time (ms) | fwd: 1386.69 | bwd: 1827.08 | bwd_inner: 1815.77 | bwd_allreduce: 10.36 | step: 21.54 15%|█▌ | 442/2865 [3:01:09<9:52:03, 14.66s/it] {'loss': 1.0142, 'learning_rate': 3.840208752605937e-05, 'epoch': 0.46} 15%|█▌ | 442/2865 [3:01:09<9:52:03, 14.66s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 164.77 | bwd_microstep: 214.40 | bwd_inner_microstep: 214.09 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.55 | bwd_microstep: 217.58 | bwd_inner_microstep: 217.41 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 162.41 | bwd_microstep: 215.25 | bwd_inner_microstep: 215.08 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.02 | bwd_microstep: 223.87 | bwd_inner_microstep: 223.68 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.68 | bwd_microstep: 218.21 | bwd_inner_microstep: 218.06 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 162.89 | bwd_microstep: 218.63 | bwd_inner_microstep: 218.48 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 162.27 | bwd_microstep: 222.26 | bwd_inner_microstep: 222.07 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 164.78 | bwd_microstep: 251.32 | bwd_inner_microstep: 241.50 | bwd_allreduce_microstep: 9.65 | step_microstep: 21.15 [Rank 0] time (ms) | fwd: 1300.25 | bwd: 1781.63 | bwd_inner: 1770.71 | bwd_allreduce: 10.21 | step: 21.71 15%|█▌ | 443/2865 [3:01:12<7:33:54, 11.24s/it] {'loss': 0.9414, 'learning_rate': 3.8393220225423526e-05, 'epoch': 0.46} 15%|█▌ | 443/2865 [3:01:12<7:33:54, 11.24s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 162.53 | bwd_microstep: 230.52 | bwd_inner_microstep: 230.15 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 162.84 | bwd_microstep: 224.38 | bwd_inner_microstep: 224.22 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 164.20 | bwd_microstep: 221.30 | bwd_inner_microstep: 220.99 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 163.69 | bwd_microstep: 222.82 | bwd_inner_microstep: 222.69 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.45 | bwd_microstep: 220.26 | bwd_inner_microstep: 220.12 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 166.41 | bwd_microstep: 248.76 | bwd_inner_microstep: 248.57 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 172.26 | bwd_microstep: 263.58 | bwd_inner_microstep: 263.17 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.15 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 175.77 | bwd_microstep: 303.30 | bwd_inner_microstep: 292.18 | bwd_allreduce_microstep: 11.00 | step_microstep: 24.41 [Rank 0] time (ms) | fwd: 1329.02 | bwd: 1935.07 | bwd_inner: 1922.46 | bwd_allreduce: 11.72 | step: 25.34 15%|█▌ | 444/2865 [3:01:16<5:59:25, 8.91s/it] {'loss': 0.9225, 'learning_rate': 3.8384329418698676e-05, 'epoch': 0.46} 15%|█▌ | 444/2865 [3:01:16<5:59:25, 8.91s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 217.85 | bwd_microstep: 256.46 | bwd_inner_microstep: 256.31 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 176.28 | bwd_microstep: 261.79 | bwd_inner_microstep: 261.63 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 176.82 | bwd_microstep: 260.91 | bwd_inner_microstep: 260.69 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 176.28 | bwd_microstep: 260.80 | bwd_inner_microstep: 260.65 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 173.57 | bwd_microstep: 261.44 | bwd_inner_microstep: 261.01 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 173.54 | bwd_microstep: 260.72 | bwd_inner_microstep: 260.36 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 176.25 | bwd_microstep: 258.05 | bwd_inner_microstep: 257.82 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.98 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.89 | bwd_microstep: 281.31 | bwd_inner_microstep: 267.69 | bwd_allreduce_microstep: 13.48 | step_microstep: 22.05 [Rank 0] time (ms) | fwd: 1431.34 | bwd: 2101.59 | bwd_inner: 2086.61 | bwd_allreduce: 14.10 | step: 22.75 16%|█▌ | 445/2865 [3:01:20<4:56:32, 7.35s/it] {'loss': 0.8286, 'learning_rate': 3.837541511724705e-05, 'epoch': 0.47} 16%|█▌ | 445/2865 [3:01:20<4:56:32, 7.35s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 952 [Rank 0] time (ms) | fwd_microstep: 179.24 | bwd_microstep: 252.73 | bwd_inner_microstep: 252.45 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.99 | bwd_microstep: 215.92 | bwd_inner_microstep: 215.73 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 163.39 | bwd_microstep: 216.39 | bwd_inner_microstep: 216.22 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 162.08 | bwd_microstep: 221.97 | bwd_inner_microstep: 221.70 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 163.78 | bwd_microstep: 218.18 | bwd_inner_microstep: 217.97 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 148.12 | bwd_microstep: 191.79 | bwd_inner_microstep: 191.66 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 160.85 | bwd_microstep: 219.58 | bwd_inner_microstep: 219.41 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.97 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.40 | bwd_microstep: 249.92 | bwd_inner_microstep: 239.88 | bwd_allreduce_microstep: 9.89 | step_microstep: 21.49 [Rank 0] time (ms) | fwd: 1301.73 | bwd: 1786.61 | bwd_inner: 1775.36 | bwd_allreduce: 10.48 | step: 22.20 16%|█▌ | 446/2865 [3:01:23<4:07:00, 6.13s/it] {'loss': 0.7936, 'learning_rate': 3.8366477332460916e-05, 'epoch': 0.47} 16%|█▌ | 446/2865 [3:01:23<4:07:00, 6.13s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 163.16 | bwd_microstep: 218.21 | bwd_inner_microstep: 217.91 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 162.32 | bwd_microstep: 218.54 | bwd_inner_microstep: 218.37 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 163.23 | bwd_microstep: 219.53 | bwd_inner_microstep: 219.33 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 164.41 | bwd_microstep: 217.57 | bwd_inner_microstep: 217.39 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.16 | bwd_microstep: 280.76 | bwd_inner_microstep: 280.59 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 184.78 | bwd_microstep: 240.12 | bwd_inner_microstep: 239.98 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.09 | bwd_microstep: 233.93 | bwd_inner_microstep: 233.61 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.35 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 177.00 | bwd_microstep: 358.36 | bwd_inner_microstep: 345.58 | bwd_allreduce_microstep: 12.63 | step_microstep: 25.08 [Rank 0] time (ms) | fwd: 1338.00 | bwd: 1987.10 | bwd_inner: 1973.15 | bwd_allreduce: 13.25 | step: 25.68 16%|█▌ | 447/2865 [3:01:26<3:35:08, 5.34s/it] {'loss': 0.7526, 'learning_rate': 3.835751607576254e-05, 'epoch': 0.47} 16%|█▌ | 447/2865 [3:01:26<3:35:08, 5.34s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 165.94 | bwd_microstep: 229.22 | bwd_inner_microstep: 228.98 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 171.57 | bwd_microstep: 263.56 | bwd_inner_microstep: 263.39 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 183.43 | bwd_microstep: 260.57 | bwd_inner_microstep: 260.40 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 175.79 | bwd_microstep: 265.28 | bwd_inner_microstep: 265.00 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 177.68 | bwd_microstep: 246.00 | bwd_inner_microstep: 245.80 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 162.22 | bwd_microstep: 216.08 | bwd_inner_microstep: 215.70 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.01 | bwd_microstep: 225.78 | bwd_inner_microstep: 225.57 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 165.17 | bwd_microstep: 250.51 | bwd_inner_microstep: 240.77 | bwd_allreduce_microstep: 9.67 | step_microstep: 21.88 [Rank 0] time (ms) | fwd: 1362.70 | bwd: 1957.13 | bwd_inner: 1945.96 | bwd_allreduce: 10.32 | step: 22.67 16%|█▌ | 448/2865 [3:01:30<3:12:48, 4.79s/it] {'loss': 1.0087, 'learning_rate': 3.834853135860419e-05, 'epoch': 0.47} 16%|█▌ | 448/2865 [3:01:30<3:12:48, 4.79s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 162.87 | bwd_microstep: 216.42 | bwd_inner_microstep: 216.11 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 162.76 | bwd_microstep: 214.90 | bwd_inner_microstep: 214.68 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 161.75 | bwd_microstep: 244.30 | bwd_inner_microstep: 244.05 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 161.78 | bwd_microstep: 240.01 | bwd_inner_microstep: 239.63 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 160.72 | bwd_microstep: 224.18 | bwd_inner_microstep: 224.10 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 162.60 | bwd_microstep: 233.08 | bwd_inner_microstep: 232.94 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 163.08 | bwd_microstep: 222.21 | bwd_inner_microstep: 221.85 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 165.73 | bwd_microstep: 305.17 | bwd_inner_microstep: 292.16 | bwd_allreduce_microstep: 12.85 | step_microstep: 21.37 [Rank 0] time (ms) | fwd: 1301.17 | bwd: 1900.41 | bwd_inner: 1885.92 | bwd_allreduce: 13.54 | step: 22.15 16%|█▌ | 449/2865 [3:01:33<2:55:36, 4.36s/it] {'loss': 0.8323, 'learning_rate': 3.833952319246812e-05, 'epoch': 0.47} 16%|█▌ | 449/2865 [3:01:33<2:55:36, 4.36s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 176.81 | bwd_microstep: 257.25 | bwd_inner_microstep: 257.10 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 177.69 | bwd_microstep: 260.03 | bwd_inner_microstep: 259.73 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 159.42 | bwd_microstep: 233.17 | bwd_inner_microstep: 232.75 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.26 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 168.33 | bwd_microstep: 267.79 | bwd_inner_microstep: 267.38 | bwd_allreduce_microstep: 0.20 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 178.58 | bwd_microstep: 274.04 | bwd_inner_microstep: 273.82 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 177.90 | bwd_microstep: 260.12 | bwd_inner_microstep: 259.85 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 162.44 | bwd_microstep: 218.47 | bwd_inner_microstep: 218.29 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.55 | bwd_microstep: 258.06 | bwd_inner_microstep: 248.33 | bwd_allreduce_microstep: 9.65 | step_microstep: 21.13 [Rank 0] time (ms) | fwd: 1364.56 | bwd: 2029.10 | bwd_inner: 2017.64 | bwd_allreduce: 10.48 | step: 22.12 16%|█▌ | 450/2865 [3:01:37<2:46:11, 4.13s/it] {'loss': 0.8661, 'learning_rate': 3.8330491588866545e-05, 'epoch': 0.47} 16%|█▌ | 450/2865 [3:01:37<2:46:11, 4.13s/it][INFO|trainer.py:3242] 2026-05-14 23:22:19,984 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 23:22:19,984 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 23:22:19,984 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-450 [INFO|configuration_utils.py:473] 2026-05-14 23:27:26,413 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-450/config.json [INFO|configuration_utils.py:594] 2026-05-14 23:27:26,421 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-450/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 23:27:31,597 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-450/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 23:27:31,616 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-450/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 23:27:31,625 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-450/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 23:27:31,627 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-450/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 23:27:40,883 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-375] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 303526.59 | bwd_microstep: 236.32 | bwd_inner_microstep: 236.10 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 163.74 | bwd_microstep: 250.90 | bwd_inner_microstep: 250.69 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 167.01 | bwd_microstep: 248.02 | bwd_inner_microstep: 247.83 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 167.95 | bwd_microstep: 277.35 | bwd_inner_microstep: 277.19 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 178.77 | bwd_microstep: 261.29 | bwd_inner_microstep: 260.87 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.24 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.42 | bwd_microstep: 259.71 | bwd_inner_microstep: 259.46 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 165.44 | bwd_microstep: 225.64 | bwd_inner_microstep: 225.23 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.01 | optimizer_gradients: 2.91 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.52 | bwd_microstep: 262.23 | bwd_inner_microstep: 251.31 | bwd_allreduce_microstep: 10.76 | step_microstep: 21.56 [Rank 0] time (ms) | fwd: 304684.06 | bwd: 2021.64 | bwd_inner: 2009.15 | bwd_allreduce: 11.51 | step: 22.43 16%|█▌ | 451/2865 [3:07:01<67:15:00, 100.29s/it] {'loss': 0.9986, 'learning_rate': 3.832143655934163e-05, 'epoch': 0.47} 16%|█▌ | 451/2865 [3:07:01<67:15:00, 100.29s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 161.14 | bwd_microstep: 225.81 | bwd_inner_microstep: 225.21 | bwd_allreduce_microstep: 0.23 | step_microstep: 0.27 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 164.66 | bwd_microstep: 231.86 | bwd_inner_microstep: 231.72 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.88 | bwd_microstep: 198.54 | bwd_inner_microstep: 198.19 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 162.41 | bwd_microstep: 231.04 | bwd_inner_microstep: 230.91 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 164.30 | bwd_microstep: 223.08 | bwd_inner_microstep: 222.76 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 165.64 | bwd_microstep: 218.89 | bwd_inner_microstep: 218.74 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 161.43 | bwd_microstep: 218.89 | bwd_inner_microstep: 218.58 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.19 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.31 | bwd_microstep: 251.28 | bwd_inner_microstep: 241.33 | bwd_allreduce_microstep: 9.86 | step_microstep: 21.69 [Rank 0] time (ms) | fwd: 1302.63 | bwd: 1799.60 | bwd_inner: 1787.81 | bwd_allreduce: 10.73 | step: 22.56 16%|█▌ | 452/2865 [3:07:05<47:42:58, 71.19s/it] {'loss': 1.0959, 'learning_rate': 3.8312358115465486e-05, 'epoch': 0.47} 16%|█▌ | 452/2865 [3:07:05<47:42:58, 71.19s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 941 [Rank 0] time (ms) | fwd_microstep: 168.40 | bwd_microstep: 222.65 | bwd_inner_microstep: 222.36 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 167.56 | bwd_microstep: 226.39 | bwd_inner_microstep: 226.01 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 161.58 | bwd_microstep: 228.27 | bwd_inner_microstep: 227.99 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 164.69 | bwd_microstep: 232.80 | bwd_inner_microstep: 232.44 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 164.26 | bwd_microstep: 228.66 | bwd_inner_microstep: 228.33 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 162.53 | bwd_microstep: 232.80 | bwd_inner_microstep: 232.42 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 162.78 | bwd_microstep: 249.73 | bwd_inner_microstep: 249.35 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.87 | bwd_microstep: 254.04 | bwd_inner_microstep: 243.64 | bwd_allreduce_microstep: 10.15 | step_microstep: 21.12 [Rank 0] time (ms) | fwd: 1314.55 | bwd: 1875.57 | bwd_inner: 1863.09 | bwd_allreduce: 11.14 | step: 22.31 16%|█▌ | 453/2865 [3:07:08<34:03:54, 50.84s/it] {'loss': 0.8215, 'learning_rate': 3.8303256268840156e-05, 'epoch': 0.47} 16%|█▌ | 453/2865 [3:07:08<34:03:54, 50.84s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 162.27 | bwd_microstep: 232.55 | bwd_inner_microstep: 232.42 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 176.48 | bwd_microstep: 223.56 | bwd_inner_microstep: 223.31 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 170.88 | bwd_microstep: 235.87 | bwd_inner_microstep: 235.51 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 161.30 | bwd_microstep: 221.73 | bwd_inner_microstep: 221.59 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.89 | bwd_microstep: 215.91 | bwd_inner_microstep: 215.68 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.92 | bwd_microstep: 240.41 | bwd_inner_microstep: 240.11 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 167.06 | bwd_microstep: 218.41 | bwd_inner_microstep: 218.20 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 164.21 | bwd_microstep: 254.18 | bwd_inner_microstep: 244.20 | bwd_allreduce_microstep: 9.90 | step_microstep: 21.29 [Rank 0] time (ms) | fwd: 1323.86 | bwd: 1842.73 | bwd_inner: 1831.38 | bwd_allreduce: 10.53 | step: 22.05 16%|█▌ | 454/2865 [3:07:11<24:30:26, 36.59s/it] {'loss': 0.8707, 'learning_rate': 3.829413103109756e-05, 'epoch': 0.47} 16%|█▌ | 454/2865 [3:07:11<24:30:26, 36.59s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.98 | bwd_microstep: 222.36 | bwd_inner_microstep: 221.98 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 162.05 | bwd_microstep: 218.84 | bwd_inner_microstep: 218.59 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 163.76 | bwd_microstep: 220.84 | bwd_inner_microstep: 220.71 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.65 | bwd_microstep: 218.76 | bwd_inner_microstep: 218.55 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 162.31 | bwd_microstep: 218.31 | bwd_inner_microstep: 218.00 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 162.30 | bwd_microstep: 229.92 | bwd_inner_microstep: 229.72 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 161.86 | bwd_microstep: 221.57 | bwd_inner_microstep: 221.29 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 177.57 | bwd_microstep: 250.80 | bwd_inner_microstep: 240.78 | bwd_allreduce_microstep: 9.86 | step_microstep: 21.23 [Rank 0] time (ms) | fwd: 1312.36 | bwd: 1801.53 | bwd_inner: 1790.10 | bwd_allreduce: 10.50 | step: 21.89 16%|█▌ | 455/2865 [3:07:15<17:48:17, 26.60s/it] {'loss': 1.0806, 'learning_rate': 3.828498241389954e-05, 'epoch': 0.48} 16%|█▌ | 455/2865 [3:07:15<17:48:17, 26.60s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 161.45 | bwd_microstep: 215.07 | bwd_inner_microstep: 214.72 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.27 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 164.51 | bwd_microstep: 218.49 | bwd_inner_microstep: 218.29 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 164.36 | bwd_microstep: 218.91 | bwd_inner_microstep: 218.77 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 165.73 | bwd_microstep: 214.94 | bwd_inner_microstep: 214.75 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 165.12 | bwd_microstep: 220.13 | bwd_inner_microstep: 219.81 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 162.03 | bwd_microstep: 214.83 | bwd_inner_microstep: 214.66 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 164.29 | bwd_microstep: 214.53 | bwd_inner_microstep: 214.39 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.93 | bwd_microstep: 248.96 | bwd_inner_microstep: 239.07 | bwd_allreduce_microstep: 9.78 | step_microstep: 21.06 [Rank 0] time (ms) | fwd: 1311.29 | bwd: 1765.98 | bwd_inner: 1754.83 | bwd_allreduce: 10.36 | step: 21.84 16%|█▌ | 456/2865 [3:07:18<13:06:42, 19.59s/it] {'loss': 0.9381, 'learning_rate': 3.8275810428937814e-05, 'epoch': 0.48} 16%|█▌ | 456/2865 [3:07:18<13:06:42, 19.59s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 938 [Rank 0] time (ms) | fwd_microstep: 162.49 | bwd_microstep: 212.38 | bwd_inner_microstep: 212.01 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 162.11 | bwd_microstep: 219.49 | bwd_inner_microstep: 219.37 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 162.86 | bwd_microstep: 216.64 | bwd_inner_microstep: 216.51 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 162.56 | bwd_microstep: 213.81 | bwd_inner_microstep: 213.43 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 161.71 | bwd_microstep: 215.87 | bwd_inner_microstep: 215.64 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.88 | bwd_microstep: 216.61 | bwd_inner_microstep: 216.23 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 164.46 | bwd_microstep: 217.95 | bwd_inner_microstep: 217.67 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.57 | bwd_microstep: 251.01 | bwd_inner_microstep: 239.30 | bwd_allreduce_microstep: 11.56 | step_microstep: 21.15 [Rank 0] time (ms) | fwd: 1298.52 | bwd: 1763.91 | bwd_inner: 1750.60 | bwd_allreduce: 12.29 | step: 21.99 16%|█▌ | 457/2865 [3:07:21<9:49:21, 14.69s/it] {'loss': 0.9628, 'learning_rate': 3.826661508793395e-05, 'epoch': 0.48} 16%|█▌ | 457/2865 [3:07:21<9:49:21, 14.69s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.89 | bwd_microstep: 220.92 | bwd_inner_microstep: 220.57 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 164.95 | bwd_microstep: 217.77 | bwd_inner_microstep: 217.42 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 163.80 | bwd_microstep: 216.14 | bwd_inner_microstep: 215.80 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.04 | bwd_microstep: 219.38 | bwd_inner_microstep: 219.15 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 163.65 | bwd_microstep: 216.27 | bwd_inner_microstep: 215.97 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 163.69 | bwd_microstep: 211.88 | bwd_inner_microstep: 211.68 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.00 | bwd_microstep: 213.59 | bwd_inner_microstep: 213.42 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.93 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 147.55 | bwd_microstep: 229.04 | bwd_inner_microstep: 218.64 | bwd_allreduce_microstep: 10.27 | step_microstep: 21.73 [Rank 0] time (ms) | fwd: 1286.44 | bwd: 1745.14 | bwd_inner: 1733.09 | bwd_allreduce: 11.07 | step: 22.45 16%|█▌ | 458/2865 [3:07:24<7:31:12, 11.25s/it] {'loss': 0.8614, 'learning_rate': 3.825739640263936e-05, 'epoch': 0.48} 16%|█▌ | 458/2865 [3:07:24<7:31:12, 11.25s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 162.26 | bwd_microstep: 216.55 | bwd_inner_microstep: 216.33 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 164.17 | bwd_microstep: 216.46 | bwd_inner_microstep: 216.24 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 169.45 | bwd_microstep: 222.00 | bwd_inner_microstep: 221.65 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 194.00 | bwd_microstep: 222.31 | bwd_inner_microstep: 222.17 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 168.26 | bwd_microstep: 218.03 | bwd_inner_microstep: 217.86 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 162.20 | bwd_microstep: 218.51 | bwd_inner_microstep: 218.33 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 147.29 | bwd_microstep: 190.68 | bwd_inner_microstep: 190.45 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 160.94 | bwd_microstep: 250.00 | bwd_inner_microstep: 240.11 | bwd_allreduce_microstep: 9.80 | step_microstep: 21.40 [Rank 0] time (ms) | fwd: 1328.45 | bwd: 1754.66 | bwd_inner: 1743.45 | bwd_allreduce: 10.41 | step: 22.05 16%|█▌ | 459/2865 [3:07:28<5:54:51, 8.85s/it] {'loss': 1.013, 'learning_rate': 3.824815438483531e-05, 'epoch': 0.48} 16%|█▌ | 459/2865 [3:07:28<5:54:51, 8.85s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 162.33 | bwd_microstep: 215.79 | bwd_inner_microstep: 215.69 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 161.03 | bwd_microstep: 220.07 | bwd_inner_microstep: 219.95 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 161.12 | bwd_microstep: 224.37 | bwd_inner_microstep: 223.97 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 162.17 | bwd_microstep: 218.39 | bwd_inner_microstep: 217.99 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 160.33 | bwd_microstep: 218.56 | bwd_inner_microstep: 218.24 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.53 | bwd_microstep: 216.78 | bwd_inner_microstep: 216.65 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 161.66 | bwd_microstep: 218.71 | bwd_inner_microstep: 218.57 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 160.39 | bwd_microstep: 253.71 | bwd_inner_microstep: 243.77 | bwd_allreduce_microstep: 9.84 | step_microstep: 21.34 [Rank 0] time (ms) | fwd: 1289.46 | bwd: 1786.51 | bwd_inner: 1775.13 | bwd_allreduce: 10.60 | step: 22.03 16%|█▌ | 460/2865 [3:07:31<4:47:16, 7.17s/it] {'loss': 0.7337, 'learning_rate': 3.823888904633288e-05, 'epoch': 0.48} 16%|█▌ | 460/2865 [3:07:31<4:47:16, 7.17s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.75 | bwd_microstep: 218.82 | bwd_inner_microstep: 218.49 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 161.28 | bwd_microstep: 217.34 | bwd_inner_microstep: 217.21 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.13 | bwd_microstep: 215.82 | bwd_inner_microstep: 215.43 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.47 | bwd_microstep: 188.58 | bwd_inner_microstep: 188.46 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 160.16 | bwd_microstep: 205.67 | bwd_inner_microstep: 205.48 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 169.13 | bwd_microstep: 238.69 | bwd_inner_microstep: 238.51 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 161.12 | bwd_microstep: 190.24 | bwd_inner_microstep: 189.88 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 161.00 | bwd_microstep: 212.89 | bwd_inner_microstep: 203.15 | bwd_allreduce_microstep: 9.66 | step_microstep: 21.02 [Rank 0] time (ms) | fwd: 1295.91 | bwd: 1688.20 | bwd_inner: 1676.99 | bwd_allreduce: 10.36 | step: 21.79 16%|█▌ | 461/2865 [3:07:34<3:58:50, 5.96s/it] {'loss': 1.0546, 'learning_rate': 3.822960039897293e-05, 'epoch': 0.48} 16%|█▌ | 461/2865 [3:07:34<3:58:50, 5.96s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 938 [Rank 0] time (ms) | fwd_microstep: 161.33 | bwd_microstep: 188.80 | bwd_inner_microstep: 188.43 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 160.57 | bwd_microstep: 188.49 | bwd_inner_microstep: 188.35 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 160.41 | bwd_microstep: 190.60 | bwd_inner_microstep: 190.23 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.62 | bwd_microstep: 187.74 | bwd_inner_microstep: 187.46 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.97 | bwd_microstep: 188.48 | bwd_inner_microstep: 188.36 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.22 | bwd_microstep: 187.66 | bwd_inner_microstep: 187.52 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.79 | bwd_microstep: 188.34 | bwd_inner_microstep: 188.07 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.56 | bwd_microstep: 212.82 | bwd_inner_microstep: 202.87 | bwd_allreduce_microstep: 9.84 | step_microstep: 21.04 [Rank 0] time (ms) | fwd: 1285.36 | bwd: 1533.06 | bwd_inner: 1521.66 | bwd_allreduce: 10.54 | step: 21.83 16%|█▌ | 462/2865 [3:07:37<3:22:48, 5.06s/it] {'loss': 1.0693, 'learning_rate': 3.822028845462615e-05, 'epoch': 0.48} 16%|█▌ | 462/2865 [3:07:37<3:22:48, 5.06s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 161.10 | bwd_microstep: 186.75 | bwd_inner_microstep: 186.63 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 161.09 | bwd_microstep: 189.42 | bwd_inner_microstep: 189.27 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.44 | bwd_microstep: 187.68 | bwd_inner_microstep: 187.36 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 161.28 | bwd_microstep: 190.98 | bwd_inner_microstep: 190.65 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.76 | bwd_microstep: 187.87 | bwd_inner_microstep: 187.58 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.46 | bwd_microstep: 189.39 | bwd_inner_microstep: 189.05 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.49 | bwd_microstep: 188.87 | bwd_inner_microstep: 188.63 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.97 | bwd_microstep: 217.75 | bwd_inner_microstep: 207.13 | bwd_allreduce_microstep: 10.47 | step_microstep: 21.16 [Rank 0] time (ms) | fwd: 1286.47 | bwd: 1538.80 | bwd_inner: 1526.79 | bwd_allreduce: 11.13 | step: 21.65 16%|█▌ | 463/2865 [3:07:40<2:57:46, 4.44s/it] {'loss': 0.9549, 'learning_rate': 3.8210953225192954e-05, 'epoch': 0.48} 16%|█▌ | 463/2865 [3:07:40<2:57:46, 4.44s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 161.25 | bwd_microstep: 189.60 | bwd_inner_microstep: 189.32 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 163.17 | bwd_microstep: 188.34 | bwd_inner_microstep: 188.21 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.72 | bwd_microstep: 190.30 | bwd_inner_microstep: 190.17 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.92 | bwd_microstep: 188.67 | bwd_inner_microstep: 188.53 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.30 | bwd_microstep: 189.85 | bwd_inner_microstep: 189.71 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.24 | bwd_microstep: 187.90 | bwd_inner_microstep: 187.62 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 147.14 | bwd_microstep: 173.34 | bwd_inner_microstep: 173.22 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.00 | bwd_microstep: 216.13 | bwd_inner_microstep: 206.46 | bwd_allreduce_microstep: 9.59 | step_microstep: 21.12 [Rank 0] time (ms) | fwd: 1274.63 | bwd: 1524.21 | bwd_inner: 1513.51 | bwd_allreduce: 10.09 | step: 21.62 16%|█▌ | 464/2865 [3:07:43<2:39:49, 3.99s/it] {'loss': 0.8109, 'learning_rate': 3.820159472260356e-05, 'epoch': 0.49} 16%|█▌ | 464/2865 [3:07:43<2:39:49, 3.99s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 938 [Rank 0] time (ms) | fwd_microstep: 162.10 | bwd_microstep: 187.96 | bwd_inner_microstep: 187.83 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 161.07 | bwd_microstep: 189.79 | bwd_inner_microstep: 189.66 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 160.91 | bwd_microstep: 188.42 | bwd_inner_microstep: 188.14 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 161.64 | bwd_microstep: 190.29 | bwd_inner_microstep: 189.99 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.78 | bwd_microstep: 188.30 | bwd_inner_microstep: 188.12 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 161.60 | bwd_microstep: 186.34 | bwd_inner_microstep: 186.27 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.29 | bwd_microstep: 191.37 | bwd_inner_microstep: 191.24 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 161.13 | bwd_microstep: 216.49 | bwd_inner_microstep: 206.74 | bwd_allreduce_microstep: 9.68 | step_microstep: 21.10 [Rank 0] time (ms) | fwd: 1289.40 | bwd: 1539.04 | bwd_inner: 1528.27 | bwd_allreduce: 10.17 | step: 21.59 16%|█▌ | 465/2865 [3:07:46<2:27:42, 3.69s/it] {'loss': 0.8778, 'learning_rate': 3.8192212958817894e-05, 'epoch': 0.49} 16%|█▌ | 465/2865 [3:07:46<2:27:42, 3.69s/it][INFO|trainer.py:3242] 2026-05-14 23:28:29,173 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 23:28:29,174 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 23:28:29,174 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-465 [INFO|configuration_utils.py:473] 2026-05-14 23:33:35,071 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-465/config.json [INFO|configuration_utils.py:594] 2026-05-14 23:33:35,080 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-465/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 23:33:40,568 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-465/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 23:33:40,589 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-465/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 23:33:40,597 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-465/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 23:33:40,609 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-465/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 23:33:50,116 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-390] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 302904.99 | bwd_microstep: 229.15 | bwd_inner_microstep: 228.85 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.74 | bwd_microstep: 222.08 | bwd_inner_microstep: 221.88 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.32 | bwd_microstep: 219.32 | bwd_inner_microstep: 219.26 | bwd_allreduce_microstep: 0.03 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 163.28 | bwd_microstep: 217.31 | bwd_inner_microstep: 217.24 | bwd_allreduce_microstep: 0.03 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.55 | bwd_microstep: 214.48 | bwd_inner_microstep: 214.41 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.47 | bwd_microstep: 215.24 | bwd_inner_microstep: 215.15 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 159.47 | bwd_microstep: 214.82 | bwd_inner_microstep: 214.71 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 159.79 | bwd_microstep: 249.24 | bwd_inner_microstep: 239.35 | bwd_allreduce_microstep: 9.82 | step_microstep: 21.17 [Rank 0] time (ms) | fwd: 304021.04 | bwd: 1781.73 | bwd_inner: 1771.01 | bwd_allreduce: 10.24 | step: 21.69 16%|█▋ | 466/2865 [3:13:10<66:35:00, 99.92s/it] {'loss': 0.7249, 'learning_rate': 3.818280794582563e-05, 'epoch': 0.49} 16%|█▋ | 466/2865 [3:13:10<66:35:00, 99.92s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 160.84 | bwd_microstep: 213.40 | bwd_inner_microstep: 213.32 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 160.97 | bwd_microstep: 217.72 | bwd_inner_microstep: 217.65 | bwd_allreduce_microstep: 0.03 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 161.20 | bwd_microstep: 218.33 | bwd_inner_microstep: 218.32 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 155.19 | bwd_microstep: 192.38 | bwd_inner_microstep: 192.36 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 154.80 | bwd_microstep: 187.67 | bwd_inner_microstep: 187.65 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 155.24 | bwd_microstep: 190.06 | bwd_inner_microstep: 190.04 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 154.91 | bwd_microstep: 189.98 | bwd_inner_microstep: 189.94 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.76 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 157.14 | bwd_microstep: 213.67 | bwd_inner_microstep: 205.70 | bwd_allreduce_microstep: 7.91 | step_microstep: 20.82 [Rank 0] time (ms) | fwd: 1260.20 | bwd: 1623.26 | bwd_inner: 1614.99 | bwd_allreduce: 8.02 | step: 21.26 16%|█▋ | 467/2865 [3:13:13<47:11:38, 70.85s/it] {'loss': 0.9543, 'learning_rate': 3.8173379695646135e-05, 'epoch': 0.49} 16%|█▋ | 467/2865 [3:13:13<47:11:38, 70.85s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 162.65 | bwd_microstep: 190.93 | bwd_inner_microstep: 190.56 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 163.86 | bwd_microstep: 191.79 | bwd_inner_microstep: 191.54 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 162.23 | bwd_microstep: 193.98 | bwd_inner_microstep: 193.83 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 162.55 | bwd_microstep: 199.67 | bwd_inner_microstep: 199.64 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 156.75 | bwd_microstep: 188.82 | bwd_inner_microstep: 188.76 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 145.72 | bwd_microstep: 182.77 | bwd_inner_microstep: 182.75 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 144.12 | bwd_microstep: 175.90 | bwd_inner_microstep: 175.87 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.76 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 155.28 | bwd_microstep: 215.53 | bwd_inner_microstep: 207.55 | bwd_allreduce_microstep: 7.91 | step_microstep: 20.81 [Rank 0] time (ms) | fwd: 1253.03 | bwd: 1539.50 | bwd_inner: 1530.70 | bwd_allreduce: 8.20 | step: 21.54 16%|█▋ | 468/2865 [3:13:16<33:36:44, 50.48s/it] {'loss': 0.7852, 'learning_rate': 3.8163928220328495e-05, 'epoch': 0.49} 16%|█▋ | 468/2865 [3:13:16<33:36:44, 50.48s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 955 [Rank 0] time (ms) | fwd_microstep: 157.98 | bwd_microstep: 194.25 | bwd_inner_microstep: 194.20 | bwd_allreduce_microstep: 0.02 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 157.82 | bwd_microstep: 190.53 | bwd_inner_microstep: 190.51 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 158.49 | bwd_microstep: 190.18 | bwd_inner_microstep: 190.16 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 156.71 | bwd_microstep: 190.81 | bwd_inner_microstep: 190.78 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 155.33 | bwd_microstep: 191.02 | bwd_inner_microstep: 191.00 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 156.21 | bwd_microstep: 194.67 | bwd_inner_microstep: 194.65 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 154.87 | bwd_microstep: 190.76 | bwd_inner_microstep: 190.72 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.01 | optimizer_gradients: 2.90 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 156.45 | bwd_microstep: 213.39 | bwd_inner_microstep: 205.62 | bwd_allreduce_microstep: 7.70 | step_microstep: 21.49 [Rank 0] time (ms) | fwd: 1253.79 | bwd: 1555.64 | bwd_inner: 1547.66 | bwd_allreduce: 7.76 | step: 21.95 16%|█▋ | 469/2865 [3:13:19<24:06:30, 36.22s/it] {'loss': 0.8454, 'learning_rate': 3.815445353195146e-05, 'epoch': 0.49} 16%|█▋ | 469/2865 [3:13:19<24:06:30, 36.22s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 155.95 | bwd_microstep: 190.02 | bwd_inner_microstep: 190.00 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 156.01 | bwd_microstep: 190.28 | bwd_inner_microstep: 190.26 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 157.99 | bwd_microstep: 188.87 | bwd_inner_microstep: 188.83 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 162.81 | bwd_microstep: 191.65 | bwd_inner_microstep: 191.28 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 163.85 | bwd_microstep: 191.56 | bwd_inner_microstep: 191.18 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 150.10 | bwd_microstep: 176.86 | bwd_inner_microstep: 176.57 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.39 | bwd_microstep: 193.15 | bwd_inner_microstep: 192.92 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 162.95 | bwd_microstep: 228.08 | bwd_inner_microstep: 218.26 | bwd_allreduce_microstep: 9.73 | step_microstep: 21.37 [Rank 0] time (ms) | fwd: 1270.94 | bwd: 1550.62 | bwd_inner: 1539.58 | bwd_allreduce: 10.30 | step: 22.14 16%|█▋ | 470/2865 [3:13:22<17:27:54, 26.25s/it] {'loss': 0.9077, 'learning_rate': 3.814495564262346e-05, 'epoch': 0.49} 16%|█▋ | 470/2865 [3:13:22<17:27:54, 26.25s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 156.42 | bwd_microstep: 189.22 | bwd_inner_microstep: 189.16 | bwd_allreduce_microstep: 0.03 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 155.89 | bwd_microstep: 190.77 | bwd_inner_microstep: 190.75 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 155.67 | bwd_microstep: 188.69 | bwd_inner_microstep: 188.67 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 156.69 | bwd_microstep: 191.51 | bwd_inner_microstep: 191.49 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 155.30 | bwd_microstep: 191.97 | bwd_inner_microstep: 191.92 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 155.75 | bwd_microstep: 188.91 | bwd_inner_microstep: 188.89 | bwd_allreduce_microstep: 0.00 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 144.17 | bwd_microstep: 188.10 | bwd_inner_microstep: 187.69 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.75 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 155.75 | bwd_microstep: 255.12 | bwd_inner_microstep: 246.97 | bwd_allreduce_microstep: 8.03 | step_microstep: 20.83 [Rank 0] time (ms) | fwd: 1235.57 | bwd: 1584.35 | bwd_inner: 1575.66 | bwd_allreduce: 8.29 | step: 21.49 16%|█▋ | 471/2865 [3:13:25<12:48:41, 19.27s/it] {'loss': 0.7896, 'learning_rate': 3.813543456448256e-05, 'epoch': 0.49} 16%|█▋ | 471/2865 [3:13:25<12:48:41, 19.27s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 156.97 | bwd_microstep: 217.97 | bwd_inner_microstep: 217.73 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 156.24 | bwd_microstep: 216.70 | bwd_inner_microstep: 216.40 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 159.42 | bwd_microstep: 220.85 | bwd_inner_microstep: 220.69 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 157.45 | bwd_microstep: 228.28 | bwd_inner_microstep: 228.02 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 155.27 | bwd_microstep: 218.54 | bwd_inner_microstep: 218.24 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 165.72 | bwd_microstep: 227.44 | bwd_inner_microstep: 227.26 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.57 | bwd_microstep: 221.70 | bwd_inner_microstep: 221.56 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.82 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.83 | bwd_microstep: 251.04 | bwd_inner_microstep: 241.28 | bwd_allreduce_microstep: 9.64 | step_microstep: 21.03 [Rank 0] time (ms) | fwd: 1273.38 | bwd: 1802.59 | bwd_inner: 1791.56 | bwd_allreduce: 10.28 | step: 21.49 16%|█▋ | 472/2865 [3:13:29<9:36:29, 14.45s/it] {'loss': 0.781, 'learning_rate': 3.812589030969647e-05, 'epoch': 0.49} 16%|█▋ | 472/2865 [3:13:29<9:36:29, 14.45s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 162.01 | bwd_microstep: 216.28 | bwd_inner_microstep: 215.91 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 161.76 | bwd_microstep: 219.51 | bwd_inner_microstep: 219.38 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 166.14 | bwd_microstep: 214.77 | bwd_inner_microstep: 214.07 | bwd_allreduce_microstep: 0.31 | step_microstep: 0.28 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 151.40 | bwd_microstep: 189.22 | bwd_inner_microstep: 188.83 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 168.40 | bwd_microstep: 219.20 | bwd_inner_microstep: 219.08 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 166.84 | bwd_microstep: 217.32 | bwd_inner_microstep: 216.93 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 166.95 | bwd_microstep: 218.15 | bwd_inner_microstep: 218.03 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 152.41 | bwd_microstep: 221.46 | bwd_inner_microstep: 211.75 | bwd_allreduce_microstep: 9.55 | step_microstep: 21.08 [Rank 0] time (ms) | fwd: 1295.81 | bwd: 1716.13 | bwd_inner: 1704.43 | bwd_allreduce: 10.51 | step: 22.11 17%|█▋ | 473/2865 [3:13:32<7:21:30, 11.07s/it] {'loss': 0.7486, 'learning_rate': 3.811632289046251e-05, 'epoch': 0.49} 17%|█▋ | 473/2865 [3:13:32<7:21:30, 11.07s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 167.56 | bwd_microstep: 214.42 | bwd_inner_microstep: 214.18 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 166.99 | bwd_microstep: 212.68 | bwd_inner_microstep: 212.55 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 165.54 | bwd_microstep: 215.47 | bwd_inner_microstep: 215.19 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 166.18 | bwd_microstep: 218.54 | bwd_inner_microstep: 218.28 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 167.49 | bwd_microstep: 212.87 | bwd_inner_microstep: 212.14 | bwd_allreduce_microstep: 0.34 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 168.07 | bwd_microstep: 215.30 | bwd_inner_microstep: 215.10 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 171.77 | bwd_microstep: 217.32 | bwd_inner_microstep: 216.96 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 168.04 | bwd_microstep: 246.26 | bwd_inner_microstep: 236.54 | bwd_allreduce_microstep: 9.63 | step_microstep: 20.96 [Rank 0] time (ms) | fwd: 1341.51 | bwd: 1753.05 | bwd_inner: 1741.43 | bwd_allreduce: 10.56 | step: 21.70 17%|█▋ | 474/2865 [3:13:35<5:48:17, 8.74s/it] {'loss': 0.7739, 'learning_rate': 3.8106732319007604e-05, 'epoch': 0.5} 17%|█▋ | 474/2865 [3:13:35<5:48:17, 8.74s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 167.94 | bwd_microstep: 216.13 | bwd_inner_microstep: 215.84 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 168.55 | bwd_microstep: 222.88 | bwd_inner_microstep: 222.50 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 166.89 | bwd_microstep: 214.83 | bwd_inner_microstep: 214.69 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 166.71 | bwd_microstep: 217.25 | bwd_inner_microstep: 216.96 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 164.99 | bwd_microstep: 215.13 | bwd_inner_microstep: 214.82 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 166.15 | bwd_microstep: 262.51 | bwd_inner_microstep: 262.35 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 187.28 | bwd_microstep: 249.51 | bwd_inner_microstep: 249.37 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.82 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 164.65 | bwd_microstep: 245.05 | bwd_inner_microstep: 235.27 | bwd_allreduce_microstep: 9.61 | step_microstep: 20.95 [Rank 0] time (ms) | fwd: 1353.01 | bwd: 1843.44 | bwd_inner: 1832.28 | bwd_allreduce: 10.21 | step: 21.64 17%|█▋ | 475/2865 [3:13:38<4:44:12, 7.13s/it] {'loss': 0.8309, 'learning_rate': 3.8097118607588294e-05, 'epoch': 0.5} 17%|█▋ | 475/2865 [3:13:38<4:44:12, 7.13s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 169.43 | bwd_microstep: 219.95 | bwd_inner_microstep: 219.81 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 169.52 | bwd_microstep: 223.07 | bwd_inner_microstep: 222.69 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 169.75 | bwd_microstep: 218.21 | bwd_inner_microstep: 218.02 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 170.34 | bwd_microstep: 213.70 | bwd_inner_microstep: 213.33 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 170.10 | bwd_microstep: 216.28 | bwd_inner_microstep: 215.98 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 166.73 | bwd_microstep: 216.45 | bwd_inner_microstep: 216.12 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 166.88 | bwd_microstep: 213.78 | bwd_inner_microstep: 213.62 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.82 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 167.75 | bwd_microstep: 258.41 | bwd_inner_microstep: 248.63 | bwd_allreduce_microstep: 9.61 | step_microstep: 20.93 [Rank 0] time (ms) | fwd: 1350.37 | bwd: 1779.98 | bwd_inner: 1768.72 | bwd_allreduce: 10.24 | step: 21.67 17%|█▋ | 476/2865 [3:13:42<3:58:30, 5.99s/it] {'loss': 0.9477, 'learning_rate': 3.808748176849065e-05, 'epoch': 0.5} 17%|█▋ | 476/2865 [3:13:42<3:58:30, 5.99s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 168.26 | bwd_microstep: 217.48 | bwd_inner_microstep: 217.20 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 171.37 | bwd_microstep: 219.31 | bwd_inner_microstep: 219.09 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 167.35 | bwd_microstep: 225.36 | bwd_inner_microstep: 225.17 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 168.48 | bwd_microstep: 218.02 | bwd_inner_microstep: 217.88 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 165.94 | bwd_microstep: 216.43 | bwd_inner_microstep: 216.11 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 166.68 | bwd_microstep: 224.52 | bwd_inner_microstep: 224.39 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 166.21 | bwd_microstep: 217.49 | bwd_inner_microstep: 217.33 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 165.02 | bwd_microstep: 246.00 | bwd_inner_microstep: 236.16 | bwd_allreduce_microstep: 9.67 | step_microstep: 20.80 [Rank 0] time (ms) | fwd: 1339.20 | bwd: 1784.69 | bwd_inner: 1773.78 | bwd_allreduce: 10.20 | step: 21.37 17%|█▋ | 477/2865 [3:13:45<3:26:14, 5.18s/it] {'loss': 0.7947, 'learning_rate': 3.807782181403032e-05, 'epoch': 0.5} 17%|█▋ | 477/2865 [3:13:45<3:26:14, 5.18s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 168.47 | bwd_microstep: 215.18 | bwd_inner_microstep: 215.06 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 166.48 | bwd_microstep: 216.13 | bwd_inner_microstep: 215.85 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 166.52 | bwd_microstep: 216.78 | bwd_inner_microstep: 216.61 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 165.83 | bwd_microstep: 216.42 | bwd_inner_microstep: 216.13 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 153.49 | bwd_microstep: 188.44 | bwd_inner_microstep: 188.31 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 167.62 | bwd_microstep: 217.32 | bwd_inner_microstep: 217.13 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 165.68 | bwd_microstep: 214.66 | bwd_inner_microstep: 214.45 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 147.84 | bwd_microstep: 223.81 | bwd_inner_microstep: 214.09 | bwd_allreduce_microstep: 9.64 | step_microstep: 20.95 [Rank 0] time (ms) | fwd: 1301.80 | bwd: 1708.83 | bwd_inner: 1698.06 | bwd_allreduce: 10.08 | step: 21.46 17%|█▋ | 478/2865 [3:13:48<3:02:21, 4.58s/it] {'loss': 0.9982, 'learning_rate': 3.80681387565525e-05, 'epoch': 0.5} 17%|█▋ | 478/2865 [3:13:48<3:02:21, 4.58s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 168.18 | bwd_microstep: 216.84 | bwd_inner_microstep: 216.71 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 165.16 | bwd_microstep: 215.26 | bwd_inner_microstep: 215.14 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 167.05 | bwd_microstep: 217.85 | bwd_inner_microstep: 217.55 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 166.49 | bwd_microstep: 219.75 | bwd_inner_microstep: 219.42 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 166.04 | bwd_microstep: 220.52 | bwd_inner_microstep: 220.35 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 167.25 | bwd_microstep: 220.49 | bwd_inner_microstep: 220.12 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.73 | bwd_microstep: 225.31 | bwd_inner_microstep: 225.19 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.43 | bwd_microstep: 264.38 | bwd_inner_microstep: 253.28 | bwd_allreduce_microstep: 10.91 | step_microstep: 21.15 [Rank 0] time (ms) | fwd: 1325.23 | bwd: 1800.51 | bwd_inner: 1788.16 | bwd_allreduce: 11.52 | step: 21.78 17%|█▋ | 479/2865 [3:13:51<2:46:46, 4.19s/it] {'loss': 0.9384, 'learning_rate': 3.805843260843189e-05, 'epoch': 0.5} 17%|█▋ | 479/2865 [3:13:51<2:46:46, 4.19s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 938 [Rank 0] time (ms) | fwd_microstep: 164.34 | bwd_microstep: 216.04 | bwd_inner_microstep: 215.91 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 165.54 | bwd_microstep: 220.82 | bwd_inner_microstep: 220.69 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.59 | bwd_microstep: 217.88 | bwd_inner_microstep: 217.75 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 161.94 | bwd_microstep: 227.72 | bwd_inner_microstep: 227.60 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 165.37 | bwd_microstep: 230.84 | bwd_inner_microstep: 230.71 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.12 | bwd_microstep: 235.83 | bwd_inner_microstep: 235.46 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 165.54 | bwd_microstep: 228.37 | bwd_inner_microstep: 228.25 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 162.57 | bwd_microstep: 262.11 | bwd_inner_microstep: 252.37 | bwd_allreduce_microstep: 9.65 | step_microstep: 20.81 [Rank 0] time (ms) | fwd: 1307.90 | bwd: 1839.72 | bwd_inner: 1828.94 | bwd_allreduce: 10.18 | step: 21.43 17%|█▋ | 480/2865 [3:13:55<2:36:51, 3.95s/it] {'loss': 0.9454, 'learning_rate': 3.804870338207273e-05, 'epoch': 0.5} 17%|█▋ | 480/2865 [3:13:55<2:36:51, 3.95s/it][INFO|trainer.py:3242] 2026-05-14 23:34:38,053 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 23:34:38,053 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 23:34:38,053 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-480 [INFO|configuration_utils.py:473] 2026-05-14 23:39:42,440 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-480/config.json [INFO|configuration_utils.py:594] 2026-05-14 23:39:42,448 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-480/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 23:39:47,714 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-480/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 23:39:47,724 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-480/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 23:39:47,731 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-480/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 23:39:47,732 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-480/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 23:39:58,527 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-405] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 301787.80 | bwd_microstep: 226.07 | bwd_inner_microstep: 225.88 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 160.23 | bwd_microstep: 224.01 | bwd_inner_microstep: 223.74 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 162.28 | bwd_microstep: 267.03 | bwd_inner_microstep: 266.89 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 162.54 | bwd_microstep: 233.38 | bwd_inner_microstep: 233.10 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 169.87 | bwd_microstep: 262.83 | bwd_inner_microstep: 262.45 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 167.23 | bwd_microstep: 262.81 | bwd_inner_microstep: 262.55 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 169.27 | bwd_microstep: 262.64 | bwd_inner_microstep: 262.48 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.52 | bwd_microstep: 248.94 | bwd_inner_microstep: 239.05 | bwd_allreduce_microstep: 9.74 | step_microstep: 21.17 [Rank 0] time (ms) | fwd: 302931.73 | bwd: 1987.83 | bwd_inner: 1976.56 | bwd_allreduce: 10.36 | step: 21.89 17%|█▋ | 481/2865 [3:19:19<66:14:40, 100.03s/it] {'loss': 0.8665, 'learning_rate': 3.803895108990872e-05, 'epoch': 0.5} 17%|█▋ | 481/2865 [3:19:19<66:14:40, 100.03s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 160.06 | bwd_microstep: 215.81 | bwd_inner_microstep: 215.67 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 160.38 | bwd_microstep: 217.59 | bwd_inner_microstep: 217.22 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 162.61 | bwd_microstep: 218.41 | bwd_inner_microstep: 218.28 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 159.36 | bwd_microstep: 215.20 | bwd_inner_microstep: 214.84 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 159.27 | bwd_microstep: 213.88 | bwd_inner_microstep: 213.76 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 159.41 | bwd_microstep: 216.82 | bwd_inner_microstep: 216.46 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 159.74 | bwd_microstep: 217.46 | bwd_inner_microstep: 217.02 | bwd_allreduce_microstep: 0.21 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 160.14 | bwd_microstep: 247.44 | bwd_inner_microstep: 237.96 | bwd_allreduce_microstep: 9.41 | step_microstep: 20.81 [Rank 0] time (ms) | fwd: 1280.85 | bwd: 1762.80 | bwd_inner: 1751.56 | bwd_allreduce: 10.23 | step: 21.76 17%|█▋ | 482/2865 [3:19:22<46:59:22, 70.99s/it] {'loss': 0.6593, 'learning_rate': 3.802917574440305e-05, 'epoch': 0.5} 17%|█▋ | 482/2865 [3:19:22<46:59:22, 70.99s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 160.66 | bwd_microstep: 214.75 | bwd_inner_microstep: 214.38 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 943 [Rank 0] time (ms) | fwd_microstep: 160.69 | bwd_microstep: 213.47 | bwd_inner_microstep: 213.34 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 159.96 | bwd_microstep: 214.26 | bwd_inner_microstep: 214.13 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 159.53 | bwd_microstep: 215.98 | bwd_inner_microstep: 215.63 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 162.63 | bwd_microstep: 218.10 | bwd_inner_microstep: 217.80 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 161.46 | bwd_microstep: 217.97 | bwd_inner_microstep: 217.77 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 160.59 | bwd_microstep: 216.26 | bwd_inner_microstep: 216.12 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.43 | bwd_microstep: 247.64 | bwd_inner_microstep: 237.93 | bwd_allreduce_microstep: 9.62 | step_microstep: 20.92 [Rank 0] time (ms) | fwd: 1285.83 | bwd: 1758.56 | bwd_inner: 1747.44 | bwd_allreduce: 10.32 | step: 21.66 17%|█▋ | 483/2865 [3:19:26<33:30:53, 50.65s/it] {'loss': 1.0062, 'learning_rate': 3.801937735804838e-05, 'epoch': 0.51} 17%|█▋ | 483/2865 [3:19:26<33:30:53, 50.65s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 161.60 | bwd_microstep: 214.11 | bwd_inner_microstep: 213.74 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 161.62 | bwd_microstep: 217.86 | bwd_inner_microstep: 217.56 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 163.36 | bwd_microstep: 216.93 | bwd_inner_microstep: 216.55 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 174.59 | bwd_microstep: 219.12 | bwd_inner_microstep: 218.68 | bwd_allreduce_microstep: 0.22 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 162.23 | bwd_microstep: 218.00 | bwd_inner_microstep: 217.61 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 161.18 | bwd_microstep: 212.73 | bwd_inner_microstep: 212.43 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.83 | bwd_microstep: 218.20 | bwd_inner_microstep: 217.84 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 164.99 | bwd_microstep: 249.08 | bwd_inner_microstep: 239.18 | bwd_allreduce_microstep: 9.73 | step_microstep: 21.29 [Rank 0] time (ms) | fwd: 1310.24 | bwd: 1766.25 | bwd_inner: 1754.14 | bwd_allreduce: 10.79 | step: 22.43 17%|█▋ | 484/2865 [3:19:29<24:07:10, 36.47s/it] {'loss': 0.7783, 'learning_rate': 3.8009555943366824e-05, 'epoch': 0.51} 17%|█▋ | 484/2865 [3:19:29<24:07:10, 36.47s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 161.28 | bwd_microstep: 214.12 | bwd_inner_microstep: 213.73 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 162.26 | bwd_microstep: 217.79 | bwd_inner_microstep: 217.42 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 163.73 | bwd_microstep: 216.24 | bwd_inner_microstep: 215.88 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 162.59 | bwd_microstep: 231.76 | bwd_inner_microstep: 231.60 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 171.78 | bwd_microstep: 254.45 | bwd_inner_microstep: 254.17 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 174.17 | bwd_microstep: 253.97 | bwd_inner_microstep: 253.77 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 171.55 | bwd_microstep: 258.08 | bwd_inner_microstep: 257.91 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.93 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 172.22 | bwd_microstep: 300.35 | bwd_inner_microstep: 289.70 | bwd_allreduce_microstep: 10.56 | step_microstep: 21.35 [Rank 0] time (ms) | fwd: 1339.44 | bwd: 1946.92 | bwd_inner: 1934.60 | bwd_allreduce: 11.39 | step: 22.30 17%|█▋ | 485/2865 [3:19:32<17:33:56, 26.57s/it] {'loss': 0.9229, 'learning_rate': 3.799971151290989e-05, 'epoch': 0.51} 17%|█▋ | 485/2865 [3:19:32<17:33:56, 26.57s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 163.26 | bwd_microstep: 215.09 | bwd_inner_microstep: 214.72 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 164.98 | bwd_microstep: 260.05 | bwd_inner_microstep: 259.63 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 173.36 | bwd_microstep: 259.39 | bwd_inner_microstep: 258.96 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 170.77 | bwd_microstep: 258.77 | bwd_inner_microstep: 258.58 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 173.96 | bwd_microstep: 261.39 | bwd_inner_microstep: 260.98 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 175.05 | bwd_microstep: 260.03 | bwd_inner_microstep: 259.74 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 178.46 | bwd_microstep: 262.28 | bwd_inner_microstep: 261.85 | bwd_allreduce_microstep: 0.20 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 160.57 | bwd_microstep: 252.42 | bwd_inner_microstep: 242.57 | bwd_allreduce_microstep: 9.77 | step_microstep: 21.02 [Rank 0] time (ms) | fwd: 1360.25 | bwd: 2029.66 | bwd_inner: 2017.55 | bwd_allreduce: 10.81 | step: 22.22 17%|█▋ | 486/2865 [3:19:36<13:00:10, 19.68s/it] {'loss': 0.8356, 'learning_rate': 3.798984407925853e-05, 'epoch': 0.51} 17%|█▋ | 486/2865 [3:19:36<13:00:10, 19.68s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 942 [Rank 0] time (ms) | fwd_microstep: 165.06 | bwd_microstep: 214.01 | bwd_inner_microstep: 213.65 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 162.42 | bwd_microstep: 219.63 | bwd_inner_microstep: 219.25 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 164.52 | bwd_microstep: 219.72 | bwd_inner_microstep: 219.41 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 166.29 | bwd_microstep: 220.43 | bwd_inner_microstep: 220.06 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 162.44 | bwd_microstep: 220.40 | bwd_inner_microstep: 220.01 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.63 | bwd_microstep: 219.38 | bwd_inner_microstep: 219.06 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 163.34 | bwd_microstep: 223.39 | bwd_inner_microstep: 223.03 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 162.28 | bwd_microstep: 250.19 | bwd_inner_microstep: 240.23 | bwd_allreduce_microstep: 9.79 | step_microstep: 20.94 [Rank 0] time (ms) | fwd: 1307.85 | bwd: 1787.32 | bwd_inner: 1775.28 | bwd_allreduce: 10.86 | step: 21.77 17%|█▋ | 487/2865 [3:19:39<9:44:46, 14.75s/it] {'loss': 0.8971, 'learning_rate': 3.7979953655023084e-05, 'epoch': 0.51} 17%|█▋ | 487/2865 [3:19:39<9:44:46, 14.75s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 162.03 | bwd_microstep: 216.09 | bwd_inner_microstep: 215.72 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 162.28 | bwd_microstep: 220.48 | bwd_inner_microstep: 220.14 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.98 | bwd_microstep: 218.33 | bwd_inner_microstep: 218.17 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 160.97 | bwd_microstep: 218.49 | bwd_inner_microstep: 218.08 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 147.11 | bwd_microstep: 191.98 | bwd_inner_microstep: 191.65 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.82 | bwd_microstep: 221.92 | bwd_inner_microstep: 221.64 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 160.85 | bwd_microstep: 217.23 | bwd_inner_microstep: 217.10 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 161.43 | bwd_microstep: 254.31 | bwd_inner_microstep: 244.32 | bwd_allreduce_microstep: 9.83 | step_microstep: 21.03 [Rank 0] time (ms) | fwd: 1276.35 | bwd: 1758.99 | bwd_inner: 1747.33 | bwd_allreduce: 10.65 | step: 21.69 17%|█▋ | 488/2865 [3:19:42<7:27:16, 11.29s/it] {'loss': 0.9609, 'learning_rate': 3.797004025284327e-05, 'epoch': 0.51} 17%|█▋ | 488/2865 [3:19:42<7:27:16, 11.29s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 164.71 | bwd_microstep: 235.89 | bwd_inner_microstep: 235.56 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 166.54 | bwd_microstep: 243.77 | bwd_inner_microstep: 243.49 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 168.71 | bwd_microstep: 253.70 | bwd_inner_microstep: 253.57 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 169.53 | bwd_microstep: 240.30 | bwd_inner_microstep: 240.15 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 168.45 | bwd_microstep: 239.46 | bwd_inner_microstep: 239.16 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 165.00 | bwd_microstep: 222.81 | bwd_inner_microstep: 222.50 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 163.30 | bwd_microstep: 224.71 | bwd_inner_microstep: 224.32 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.00 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 168.53 | bwd_microstep: 260.50 | bwd_inner_microstep: 246.95 | bwd_allreduce_microstep: 13.37 | step_microstep: 21.44 [Rank 0] time (ms) | fwd: 1334.59 | bwd: 1921.26 | bwd_inner: 1906.18 | bwd_allreduce: 14.07 | step: 22.13 17%|█▋ | 489/2865 [3:19:46<5:53:57, 8.94s/it] {'loss': 0.8788, 'learning_rate': 3.796010388538819e-05, 'epoch': 0.51} 17%|█▋ | 489/2865 [3:19:46<5:53:57, 8.94s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 181.95 | bwd_microstep: 262.79 | bwd_inner_microstep: 262.52 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 188.27 | bwd_microstep: 261.21 | bwd_inner_microstep: 260.77 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 185.95 | bwd_microstep: 231.81 | bwd_inner_microstep: 231.25 | bwd_allreduce_microstep: 0.22 | step_microstep: 0.29 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 167.61 | bwd_microstep: 254.74 | bwd_inner_microstep: 254.32 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 174.43 | bwd_microstep: 233.79 | bwd_inner_microstep: 233.24 | bwd_allreduce_microstep: 0.23 | step_microstep: 0.29 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 169.85 | bwd_microstep: 280.95 | bwd_inner_microstep: 280.55 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 166.38 | bwd_microstep: 269.20 | bwd_inner_microstep: 268.96 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.04 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 151.12 | bwd_microstep: 290.91 | bwd_inner_microstep: 278.48 | bwd_allreduce_microstep: 12.26 | step_microstep: 22.34 [Rank 0] time (ms) | fwd: 1385.41 | bwd: 2085.70 | bwd_inner: 2070.78 | bwd_allreduce: 13.37 | step: 23.71 17%|█▋ | 490/2865 [3:19:50<4:51:21, 7.36s/it] {'loss': 0.9698, 'learning_rate': 3.795014456535627e-05, 'epoch': 0.51} 17%|█▋ | 490/2865 [3:19:50<4:51:21, 7.36s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 174.53 | bwd_microstep: 267.54 | bwd_inner_microstep: 267.11 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 175.75 | bwd_microstep: 303.96 | bwd_inner_microstep: 303.58 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 173.28 | bwd_microstep: 264.84 | bwd_inner_microstep: 264.65 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 172.71 | bwd_microstep: 270.97 | bwd_inner_microstep: 270.72 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 177.72 | bwd_microstep: 250.14 | bwd_inner_microstep: 249.87 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 176.74 | bwd_microstep: 271.86 | bwd_inner_microstep: 271.70 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 174.16 | bwd_microstep: 275.32 | bwd_inner_microstep: 275.15 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.33 | bwd_microstep: 276.69 | bwd_inner_microstep: 265.96 | bwd_allreduce_microstep: 10.61 | step_microstep: 21.28 [Rank 0] time (ms) | fwd: 1385.11 | bwd: 2181.43 | bwd_inner: 2169.16 | bwd_allreduce: 11.40 | step: 22.06 17%|█▋ | 491/2865 [3:19:53<4:08:36, 6.28s/it] {'loss': 0.7551, 'learning_rate': 3.794016230547527e-05, 'epoch': 0.51} 17%|█▋ | 491/2865 [3:19:53<4:08:36, 6.28s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 175.44 | bwd_microstep: 263.26 | bwd_inner_microstep: 263.11 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 174.70 | bwd_microstep: 260.86 | bwd_inner_microstep: 260.23 | bwd_allreduce_microstep: 0.27 | step_microstep: 0.29 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 175.41 | bwd_microstep: 266.06 | bwd_inner_microstep: 265.89 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 178.60 | bwd_microstep: 268.33 | bwd_inner_microstep: 268.17 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 173.86 | bwd_microstep: 257.18 | bwd_inner_microstep: 257.01 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 178.34 | bwd_microstep: 270.41 | bwd_inner_microstep: 269.80 | bwd_allreduce_microstep: 0.26 | step_microstep: 0.26 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 909 [Rank 0] time (ms) | fwd_microstep: 177.41 | bwd_microstep: 266.23 | bwd_inner_microstep: 265.83 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.96 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 157.91 | bwd_microstep: 273.38 | bwd_inner_microstep: 262.59 | bwd_allreduce_microstep: 10.60 | step_microstep: 21.40 [Rank 0] time (ms) | fwd: 1391.54 | bwd: 2125.93 | bwd_inner: 2113.14 | bwd_allreduce: 11.61 | step: 22.48 17%|█▋ | 492/2865 [3:19:57<3:38:01, 5.51s/it] {'loss': 1.1267, 'learning_rate': 3.793015711850229e-05, 'epoch': 0.51} 17%|█▋ | 492/2865 [3:19:57<3:38:01, 5.51s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 171.86 | bwd_microstep: 256.22 | bwd_inner_microstep: 256.06 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 174.70 | bwd_microstep: 273.26 | bwd_inner_microstep: 273.10 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 166.32 | bwd_microstep: 302.37 | bwd_inner_microstep: 302.12 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 165.10 | bwd_microstep: 224.82 | bwd_inner_microstep: 224.55 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 165.97 | bwd_microstep: 254.24 | bwd_inner_microstep: 253.96 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 164.13 | bwd_microstep: 235.17 | bwd_inner_microstep: 234.84 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 163.58 | bwd_microstep: 228.53 | bwd_inner_microstep: 228.24 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 166.90 | bwd_microstep: 281.83 | bwd_inner_microstep: 271.60 | bwd_allreduce_microstep: 10.07 | step_microstep: 20.93 [Rank 0] time (ms) | fwd: 1338.43 | bwd: 2056.56 | bwd_inner: 2044.92 | bwd_allreduce: 10.75 | step: 21.61 17%|█▋ | 493/2865 [3:20:01<3:14:58, 4.93s/it] {'loss': 0.9498, 'learning_rate': 3.792012901722371e-05, 'epoch': 0.52} 17%|█▋ | 493/2865 [3:20:01<3:14:58, 4.93s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 166.95 | bwd_microstep: 228.20 | bwd_inner_microstep: 227.82 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 179.74 | bwd_microstep: 226.99 | bwd_inner_microstep: 226.65 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 163.76 | bwd_microstep: 224.46 | bwd_inner_microstep: 224.25 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 162.39 | bwd_microstep: 223.18 | bwd_inner_microstep: 222.80 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 161.00 | bwd_microstep: 227.86 | bwd_inner_microstep: 227.46 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 164.44 | bwd_microstep: 231.91 | bwd_inner_microstep: 231.71 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 163.96 | bwd_microstep: 240.52 | bwd_inner_microstep: 240.39 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.01 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.14 | bwd_microstep: 256.59 | bwd_inner_microstep: 245.34 | bwd_allreduce_microstep: 11.16 | step_microstep: 22.03 [Rank 0] time (ms) | fwd: 1323.23 | bwd: 1859.87 | bwd_inner: 1846.84 | bwd_allreduce: 12.04 | step: 22.70 17%|█▋ | 494/2865 [3:20:04<2:56:25, 4.46s/it] {'loss': 0.9095, 'learning_rate': 3.791007801445521e-05, 'epoch': 0.52} 17%|█▋ | 494/2865 [3:20:04<2:56:25, 4.46s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 941 [Rank 0] time (ms) | fwd_microstep: 164.41 | bwd_microstep: 221.00 | bwd_inner_microstep: 220.86 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 163.16 | bwd_microstep: 226.10 | bwd_inner_microstep: 225.77 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 162.73 | bwd_microstep: 223.61 | bwd_inner_microstep: 223.45 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 163.05 | bwd_microstep: 224.88 | bwd_inner_microstep: 224.58 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 164.00 | bwd_microstep: 229.03 | bwd_inner_microstep: 228.73 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 161.83 | bwd_microstep: 226.06 | bwd_inner_microstep: 225.77 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 162.70 | bwd_microstep: 215.64 | bwd_inner_microstep: 215.36 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.06 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 169.11 | bwd_microstep: 251.03 | bwd_inner_microstep: 240.95 | bwd_allreduce_microstep: 9.91 | step_microstep: 21.22 [Rank 0] time (ms) | fwd: 1310.83 | bwd: 1817.44 | bwd_inner: 1805.98 | bwd_allreduce: 10.61 | step: 21.80 17%|█▋ | 495/2865 [3:20:07<2:42:47, 4.12s/it] {'loss': 0.9123, 'learning_rate': 3.7900004123041704e-05, 'epoch': 0.52} 17%|█▋ | 495/2865 [3:20:07<2:42:47, 4.12s/it][INFO|trainer.py:3242] 2026-05-14 23:40:50,511 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 23:40:50,511 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 23:40:50,511 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-495 [INFO|configuration_utils.py:473] 2026-05-14 23:45:56,621 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-495/config.json [INFO|configuration_utils.py:594] 2026-05-14 23:45:56,628 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-495/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 23:46:02,126 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-495/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 23:46:02,136 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-495/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 23:46:02,143 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-495/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 23:46:02,144 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-495/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 23:46:12,060 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-420] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 303041.58 | bwd_microstep: 221.15 | bwd_inner_microstep: 220.84 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 938 [Rank 0] time (ms) | fwd_microstep: 161.46 | bwd_microstep: 219.99 | bwd_inner_microstep: 219.68 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.69 | bwd_microstep: 216.06 | bwd_inner_microstep: 215.93 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.39 | bwd_microstep: 224.19 | bwd_inner_microstep: 223.86 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.91 | bwd_microstep: 218.24 | bwd_inner_microstep: 218.11 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.85 | bwd_microstep: 225.27 | bwd_inner_microstep: 225.14 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.67 | bwd_microstep: 222.41 | bwd_inner_microstep: 222.03 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.97 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.90 | bwd_microstep: 255.85 | bwd_inner_microstep: 241.37 | bwd_allreduce_microstep: 14.36 | step_microstep: 21.65 [Rank 0] time (ms) | fwd: 304162.32 | bwd: 1803.33 | bwd_inner: 1787.26 | bwd_allreduce: 15.21 | step: 22.52 17%|█▋ | 496/2865 [3:25:32<66:05:32, 100.44s/it] {'loss': 0.8464, 'learning_rate': 3.788990735585741e-05, 'epoch': 0.52} 17%|█▋ | 496/2865 [3:25:32<66:05:32, 100.44s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 163.06 | bwd_microstep: 222.36 | bwd_inner_microstep: 221.94 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 162.03 | bwd_microstep: 219.48 | bwd_inner_microstep: 219.12 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 162.83 | bwd_microstep: 223.07 | bwd_inner_microstep: 222.84 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 163.99 | bwd_microstep: 221.88 | bwd_inner_microstep: 221.52 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 163.54 | bwd_microstep: 218.27 | bwd_inner_microstep: 217.89 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 163.54 | bwd_microstep: 233.76 | bwd_inner_microstep: 233.39 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 163.77 | bwd_microstep: 221.73 | bwd_inner_microstep: 221.59 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.93 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 165.59 | bwd_microstep: 257.82 | bwd_inner_microstep: 246.15 | bwd_allreduce_microstep: 11.56 | step_microstep: 21.19 [Rank 0] time (ms) | fwd: 1308.23 | bwd: 1818.60 | bwd_inner: 1804.89 | bwd_allreduce: 12.51 | step: 22.29 17%|█▋ | 497/2865 [3:25:36<46:53:45, 71.29s/it] {'loss': 0.9603, 'learning_rate': 3.787978772580573e-05, 'epoch': 0.52} 17%|█▋ | 497/2865 [3:25:36<46:53:45, 71.29s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 168.86 | bwd_microstep: 259.38 | bwd_inner_microstep: 259.23 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 173.11 | bwd_microstep: 260.66 | bwd_inner_microstep: 260.52 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 175.82 | bwd_microstep: 252.98 | bwd_inner_microstep: 252.86 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 165.51 | bwd_microstep: 265.19 | bwd_inner_microstep: 265.01 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 169.31 | bwd_microstep: 257.92 | bwd_inner_microstep: 257.53 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 175.49 | bwd_microstep: 259.42 | bwd_inner_microstep: 259.02 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 175.87 | bwd_microstep: 268.23 | bwd_inner_microstep: 267.94 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.93 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 176.25 | bwd_microstep: 307.54 | bwd_inner_microstep: 296.56 | bwd_allreduce_microstep: 10.89 | step_microstep: 21.36 [Rank 0] time (ms) | fwd: 1380.09 | bwd: 2131.46 | bwd_inner: 2119.05 | bwd_allreduce: 11.60 | step: 22.22 17%|█▋ | 498/2865 [3:25:39<33:32:42, 51.02s/it] {'loss': 0.9754, 'learning_rate': 3.786964524581931e-05, 'epoch': 0.52} 17%|█▋ | 498/2865 [3:25:39<33:32:42, 51.02s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 940 [Rank 0] time (ms) | fwd_microstep: 174.87 | bwd_microstep: 259.82 | bwd_inner_microstep: 259.44 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 947 [Rank 0] time (ms) | fwd_microstep: 174.22 | bwd_microstep: 265.68 | bwd_inner_microstep: 265.27 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 176.80 | bwd_microstep: 261.81 | bwd_inner_microstep: 261.65 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 171.73 | bwd_microstep: 267.46 | bwd_inner_microstep: 267.30 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.43 | bwd_microstep: 219.19 | bwd_inner_microstep: 218.96 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 163.71 | bwd_microstep: 217.83 | bwd_inner_microstep: 217.46 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 163.17 | bwd_microstep: 227.27 | bwd_inner_microstep: 226.97 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.95 | bwd_microstep: 253.76 | bwd_inner_microstep: 243.90 | bwd_allreduce_microstep: 9.71 | step_microstep: 20.79 [Rank 0] time (ms) | fwd: 1348.75 | bwd: 1972.99 | bwd_inner: 1961.45 | bwd_allreduce: 10.50 | step: 21.76 17%|█▋ | 499/2865 [3:25:43<24:09:53, 36.77s/it] {'loss': 0.8308, 'learning_rate': 3.7859479928860004e-05, 'epoch': 0.52} 17%|█▋ | 499/2865 [3:25:43<24:09:53, 36.77s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 162.45 | bwd_microstep: 222.98 | bwd_inner_microstep: 222.84 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 163.78 | bwd_microstep: 219.15 | bwd_inner_microstep: 218.98 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 163.45 | bwd_microstep: 222.72 | bwd_inner_microstep: 222.56 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 162.73 | bwd_microstep: 221.45 | bwd_inner_microstep: 221.32 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 164.74 | bwd_microstep: 221.89 | bwd_inner_microstep: 221.51 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 166.26 | bwd_microstep: 234.97 | bwd_inner_microstep: 234.80 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 151.40 | bwd_microstep: 201.69 | bwd_inner_microstep: 201.40 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.29 | bwd_microstep: 267.75 | bwd_inner_microstep: 257.75 | bwd_allreduce_microstep: 9.85 | step_microstep: 21.28 [Rank 0] time (ms) | fwd: 1297.96 | bwd: 1812.76 | bwd_inner: 1801.52 | bwd_allreduce: 10.47 | step: 22.06 17%|█▋ | 500/2865 [3:25:46<17:33:16, 26.72s/it] {'loss': 0.9541, 'learning_rate': 3.784929178791883e-05, 'epoch': 0.52} 17%|█▋ | 500/2865 [3:25:46<17:33:16, 26.72s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 165.53 | bwd_microstep: 217.82 | bwd_inner_microstep: 217.64 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 163.36 | bwd_microstep: 218.88 | bwd_inner_microstep: 218.67 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 164.63 | bwd_microstep: 220.47 | bwd_inner_microstep: 220.32 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 163.26 | bwd_microstep: 223.66 | bwd_inner_microstep: 223.49 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 162.69 | bwd_microstep: 221.33 | bwd_inner_microstep: 221.06 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 163.14 | bwd_microstep: 224.59 | bwd_inner_microstep: 224.43 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 148.95 | bwd_microstep: 200.78 | bwd_inner_microstep: 200.56 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 164.49 | bwd_microstep: 250.91 | bwd_inner_microstep: 241.04 | bwd_allreduce_microstep: 9.75 | step_microstep: 20.83 [Rank 0] time (ms) | fwd: 1295.94 | bwd: 1778.54 | bwd_inner: 1767.57 | bwd_allreduce: 10.28 | step: 21.37 17%|█▋ | 501/2865 [3:25:50<12:55:07, 19.67s/it] {'loss': 0.9817, 'learning_rate': 3.783908083601598e-05, 'epoch': 0.52} 17%|█▋ | 501/2865 [3:25:50<12:55:07, 19.67s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 947 [Rank 0] time (ms) | fwd_microstep: 164.21 | bwd_microstep: 222.18 | bwd_inner_microstep: 221.92 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 162.31 | bwd_microstep: 224.97 | bwd_inner_microstep: 224.91 | bwd_allreduce_microstep: 0.03 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 163.46 | bwd_microstep: 224.90 | bwd_inner_microstep: 224.82 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 162.16 | bwd_microstep: 221.02 | bwd_inner_microstep: 220.96 | bwd_allreduce_microstep: 0.03 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 163.30 | bwd_microstep: 218.59 | bwd_inner_microstep: 218.52 | bwd_allreduce_microstep: 0.03 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 170.69 | bwd_microstep: 284.93 | bwd_inner_microstep: 284.73 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 154.35 | bwd_microstep: 250.82 | bwd_inner_microstep: 250.62 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 909 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.00 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 191.01 | bwd_microstep: 313.60 | bwd_inner_microstep: 301.87 | bwd_allreduce_microstep: 11.61 | step_microstep: 21.85 [Rank 0] time (ms) | fwd: 1331.34 | bwd: 1961.11 | bwd_inner: 1948.56 | bwd_allreduce: 12.02 | step: 22.41 18%|█▊ | 502/2865 [3:25:53<9:43:30, 14.82s/it] {'loss': 0.7953, 'learning_rate': 3.7828847086200816e-05, 'epoch': 0.53} 18%|█▊ | 502/2865 [3:25:53<9:43:30, 14.82s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 167.38 | bwd_microstep: 224.05 | bwd_inner_microstep: 223.77 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 170.67 | bwd_microstep: 272.23 | bwd_inner_microstep: 271.88 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 160.58 | bwd_microstep: 223.40 | bwd_inner_microstep: 223.09 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 161.71 | bwd_microstep: 224.16 | bwd_inner_microstep: 223.83 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 163.53 | bwd_microstep: 226.39 | bwd_inner_microstep: 226.14 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 162.65 | bwd_microstep: 225.01 | bwd_inner_microstep: 224.88 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 161.57 | bwd_microstep: 221.63 | bwd_inner_microstep: 221.42 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 146.77 | bwd_microstep: 228.00 | bwd_inner_microstep: 218.09 | bwd_allreduce_microstep: 9.82 | step_microstep: 21.09 [Rank 0] time (ms) | fwd: 1294.75 | bwd: 1845.03 | bwd_inner: 1833.46 | bwd_allreduce: 10.65 | step: 21.95 18%|█▊ | 503/2865 [3:25:56<7:27:16, 11.36s/it] {'loss': 1.0135, 'learning_rate': 3.781859055155182e-05, 'epoch': 0.53} 18%|█▊ | 503/2865 [3:25:56<7:27:16, 11.36s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 955 [Rank 0] time (ms) | fwd_microstep: 175.06 | bwd_microstep: 266.84 | bwd_inner_microstep: 266.67 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 174.97 | bwd_microstep: 269.25 | bwd_inner_microstep: 268.98 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 191.86 | bwd_microstep: 262.02 | bwd_inner_microstep: 261.81 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 185.69 | bwd_microstep: 273.63 | bwd_inner_microstep: 273.43 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 186.66 | bwd_microstep: 268.27 | bwd_inner_microstep: 268.01 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 167.57 | bwd_microstep: 224.74 | bwd_inner_microstep: 224.44 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.27 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 163.64 | bwd_microstep: 226.10 | bwd_inner_microstep: 225.96 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 148.20 | bwd_microstep: 232.05 | bwd_inner_microstep: 220.62 | bwd_allreduce_microstep: 11.32 | step_microstep: 21.52 [Rank 0] time (ms) | fwd: 1393.51 | bwd: 2023.02 | bwd_inner: 2010.29 | bwd_allreduce: 11.92 | step: 22.33 18%|█▊ | 504/2865 [3:26:00<5:55:23, 9.03s/it] {'loss': 0.9914, 'learning_rate': 3.7808311245176585e-05, 'epoch': 0.53} 18%|█▊ | 504/2865 [3:26:00<5:55:23, 9.03s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 164.72 | bwd_microstep: 221.78 | bwd_inner_microstep: 221.64 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 165.40 | bwd_microstep: 222.84 | bwd_inner_microstep: 222.69 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 165.54 | bwd_microstep: 220.94 | bwd_inner_microstep: 220.66 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 163.32 | bwd_microstep: 220.20 | bwd_inner_microstep: 220.01 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 165.50 | bwd_microstep: 220.18 | bwd_inner_microstep: 220.09 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 149.56 | bwd_microstep: 195.38 | bwd_inner_microstep: 195.25 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 147.62 | bwd_microstep: 189.48 | bwd_inner_microstep: 189.28 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.61 | bwd_microstep: 259.21 | bwd_inner_microstep: 246.21 | bwd_allreduce_microstep: 12.90 | step_microstep: 21.33 [Rank 0] time (ms) | fwd: 1283.17 | bwd: 1750.09 | bwd_inner: 1736.14 | bwd_allreduce: 13.33 | step: 21.94 18%|█▊ | 505/2865 [3:26:03<4:46:14, 7.28s/it] {'loss': 0.8723, 'learning_rate': 3.779800918021182e-05, 'epoch': 0.53} 18%|█▊ | 505/2865 [3:26:03<4:46:14, 7.28s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 941 [Rank 0] time (ms) | fwd_microstep: 162.37 | bwd_microstep: 218.26 | bwd_inner_microstep: 218.17 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.37 | bwd_microstep: 220.11 | bwd_inner_microstep: 220.04 | bwd_allreduce_microstep: 0.03 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.83 | bwd_microstep: 217.79 | bwd_inner_microstep: 217.72 | bwd_allreduce_microstep: 0.03 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 160.68 | bwd_microstep: 222.52 | bwd_inner_microstep: 222.36 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 160.60 | bwd_microstep: 217.28 | bwd_inner_microstep: 217.21 | bwd_allreduce_microstep: 0.03 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 162.25 | bwd_microstep: 217.97 | bwd_inner_microstep: 217.78 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 162.15 | bwd_microstep: 223.89 | bwd_inner_microstep: 223.73 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.09 | bwd_microstep: 251.49 | bwd_inner_microstep: 241.76 | bwd_allreduce_microstep: 9.63 | step_microstep: 20.92 [Rank 0] time (ms) | fwd: 1292.24 | bwd: 1789.38 | bwd_inner: 1778.89 | bwd_allreduce: 10.05 | step: 21.47 18%|█▊ | 506/2865 [3:26:06<3:58:26, 6.06s/it] {'loss': 0.9673, 'learning_rate': 3.778768436982331e-05, 'epoch': 0.53} 18%|█▊ | 506/2865 [3:26:06<3:58:26, 6.06s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 163.16 | bwd_microstep: 220.94 | bwd_inner_microstep: 220.81 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 163.43 | bwd_microstep: 219.74 | bwd_inner_microstep: 219.51 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 163.03 | bwd_microstep: 224.17 | bwd_inner_microstep: 224.05 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 162.25 | bwd_microstep: 243.89 | bwd_inner_microstep: 243.73 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.13 | bwd_microstep: 223.20 | bwd_inner_microstep: 222.83 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 166.02 | bwd_microstep: 218.93 | bwd_inner_microstep: 218.72 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.60 | bwd_microstep: 220.89 | bwd_inner_microstep: 220.46 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.79 | bwd_microstep: 248.92 | bwd_inner_microstep: 238.88 | bwd_allreduce_microstep: 9.87 | step_microstep: 20.80 [Rank 0] time (ms) | fwd: 1305.31 | bwd: 1820.82 | bwd_inner: 1809.36 | bwd_allreduce: 10.57 | step: 21.55 18%|█▊ | 507/2865 [3:26:10<3:25:33, 5.23s/it] {'loss': 0.9691, 'learning_rate': 3.7777336827205925e-05, 'epoch': 0.53} 18%|█▊ | 507/2865 [3:26:10<3:25:33, 5.23s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 163.35 | bwd_microstep: 218.56 | bwd_inner_microstep: 218.19 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 946 [Rank 0] time (ms) | fwd_microstep: 164.24 | bwd_microstep: 219.33 | bwd_inner_microstep: 219.12 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 162.64 | bwd_microstep: 222.29 | bwd_inner_microstep: 222.11 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 163.66 | bwd_microstep: 220.51 | bwd_inner_microstep: 220.34 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 163.53 | bwd_microstep: 231.58 | bwd_inner_microstep: 231.42 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 175.66 | bwd_microstep: 219.12 | bwd_inner_microstep: 218.84 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 162.01 | bwd_microstep: 216.43 | bwd_inner_microstep: 216.13 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.26 | bwd_microstep: 250.58 | bwd_inner_microstep: 240.08 | bwd_allreduce_microstep: 10.34 | step_microstep: 20.89 [Rank 0] time (ms) | fwd: 1317.24 | bwd: 1798.52 | bwd_inner: 1786.63 | bwd_allreduce: 11.03 | step: 21.47 18%|█▊ | 508/2865 [3:26:13<3:02:34, 4.65s/it] {'loss': 0.9857, 'learning_rate': 3.7766966565583557e-05, 'epoch': 0.53} 18%|█▊ | 508/2865 [3:26:13<3:02:34, 4.65s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 938 [Rank 0] time (ms) | fwd_microstep: 163.63 | bwd_microstep: 221.94 | bwd_inner_microstep: 221.58 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 162.58 | bwd_microstep: 219.60 | bwd_inner_microstep: 219.47 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 162.92 | bwd_microstep: 226.13 | bwd_inner_microstep: 225.85 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 162.38 | bwd_microstep: 224.22 | bwd_inner_microstep: 224.10 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 146.72 | bwd_microstep: 198.31 | bwd_inner_microstep: 198.16 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.79 | bwd_microstep: 223.03 | bwd_inner_microstep: 222.63 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 161.84 | bwd_microstep: 216.13 | bwd_inner_microstep: 215.96 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.86 | bwd_microstep: 248.94 | bwd_inner_microstep: 239.26 | bwd_allreduce_microstep: 9.60 | step_microstep: 21.17 [Rank 0] time (ms) | fwd: 1284.61 | bwd: 1778.44 | bwd_inner: 1767.35 | bwd_allreduce: 10.29 | step: 21.97 18%|█▊ | 509/2865 [3:26:16<2:45:37, 4.22s/it] {'loss': 0.7649, 'learning_rate': 3.775657359820916e-05, 'epoch': 0.53} 18%|█▊ | 509/2865 [3:26:16<2:45:37, 4.22s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 952 [Rank 0] time (ms) | fwd_microstep: 164.86 | bwd_microstep: 219.10 | bwd_inner_microstep: 218.75 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 163.22 | bwd_microstep: 221.76 | bwd_inner_microstep: 221.43 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 163.95 | bwd_microstep: 221.92 | bwd_inner_microstep: 221.67 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 163.74 | bwd_microstep: 219.72 | bwd_inner_microstep: 219.35 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.02 | bwd_microstep: 222.17 | bwd_inner_microstep: 222.04 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.38 | bwd_microstep: 221.37 | bwd_inner_microstep: 221.14 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 161.73 | bwd_microstep: 219.23 | bwd_inner_microstep: 218.93 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.29 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 161.40 | bwd_microstep: 252.21 | bwd_inner_microstep: 242.54 | bwd_allreduce_microstep: 9.53 | step_microstep: 21.95 [Rank 0] time (ms) | fwd: 1303.19 | bwd: 1797.65 | bwd_inner: 1786.31 | bwd_allreduce: 10.32 | step: 22.83 18%|█▊ | 510/2865 [3:26:19<2:34:11, 3.93s/it] {'loss': 0.9766, 'learning_rate': 3.774615793836469e-05, 'epoch': 0.53} 18%|█▊ | 510/2865 [3:26:19<2:34:11, 3.93s/it][INFO|trainer.py:3242] 2026-05-14 23:47:02,545 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 23:47:02,546 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 23:47:02,546 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-510 [INFO|configuration_utils.py:473] 2026-05-14 23:52:08,241 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-510/config.json [INFO|configuration_utils.py:594] 2026-05-14 23:52:08,249 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-510/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 23:52:13,888 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-510/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 23:52:13,898 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-510/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 23:52:13,913 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-510/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 23:52:13,915 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-510/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 23:52:24,136 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-435] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 302683.18 | bwd_microstep: 260.86 | bwd_inner_microstep: 260.59 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 170.34 | bwd_microstep: 262.26 | bwd_inner_microstep: 262.11 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 170.62 | bwd_microstep: 272.50 | bwd_inner_microstep: 272.15 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 169.58 | bwd_microstep: 261.70 | bwd_inner_microstep: 261.38 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 172.26 | bwd_microstep: 266.87 | bwd_inner_microstep: 266.65 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 173.47 | bwd_microstep: 269.41 | bwd_inner_microstep: 269.00 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 172.86 | bwd_microstep: 276.69 | bwd_inner_microstep: 276.38 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 159.94 | bwd_microstep: 251.74 | bwd_inner_microstep: 240.72 | bwd_allreduce_microstep: 10.87 | step_microstep: 21.11 [Rank 0] time (ms) | fwd: 303861.65 | bwd: 2122.16 | bwd_inner: 2109.43 | bwd_allreduce: 11.75 | step: 22.00 18%|█▊ | 511/2865 [3:31:45<65:44:47, 100.55s/it] {'loss': 0.97, 'learning_rate': 3.773571959936111e-05, 'epoch': 0.53} 18%|█▊ | 511/2865 [3:31:45<65:44:47, 100.55s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 944 [Rank 0] time (ms) | fwd_microstep: 162.18 | bwd_microstep: 217.86 | bwd_inner_microstep: 217.73 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.98 | bwd_microstep: 223.07 | bwd_inner_microstep: 222.85 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.39 | bwd_microstep: 220.15 | bwd_inner_microstep: 219.97 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 160.86 | bwd_microstep: 220.06 | bwd_inner_microstep: 219.71 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.59 | bwd_microstep: 217.52 | bwd_inner_microstep: 217.22 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 163.26 | bwd_microstep: 223.54 | bwd_inner_microstep: 223.14 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.22 | bwd_microstep: 217.25 | bwd_inner_microstep: 217.05 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.76 | bwd_microstep: 221.48 | bwd_inner_microstep: 211.61 | bwd_allreduce_microstep: 9.80 | step_microstep: 21.07 [Rank 0] time (ms) | fwd: 1290.13 | bwd: 1761.03 | bwd_inner: 1749.73 | bwd_allreduce: 10.45 | step: 21.74 18%|█▊ | 512/2865 [3:31:49<46:37:55, 71.35s/it] {'loss': 0.875, 'learning_rate': 3.7725258594538356e-05, 'epoch': 0.54} 18%|█▊ | 512/2865 [3:31:49<46:37:55, 71.35s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 161.75 | bwd_microstep: 190.10 | bwd_inner_microstep: 189.97 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 161.29 | bwd_microstep: 191.13 | bwd_inner_microstep: 190.78 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 161.10 | bwd_microstep: 190.91 | bwd_inner_microstep: 190.68 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 162.01 | bwd_microstep: 188.87 | bwd_inner_microstep: 188.75 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 160.20 | bwd_microstep: 189.92 | bwd_inner_microstep: 189.60 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.51 | bwd_microstep: 219.50 | bwd_inner_microstep: 219.25 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 162.41 | bwd_microstep: 217.70 | bwd_inner_microstep: 217.57 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 162.07 | bwd_microstep: 251.44 | bwd_inner_microstep: 241.51 | bwd_allreduce_microstep: 9.77 | step_microstep: 21.02 [Rank 0] time (ms) | fwd: 1292.24 | bwd: 1639.67 | bwd_inner: 1628.59 | bwd_allreduce: 10.27 | step: 21.65 18%|█▊ | 513/2865 [3:31:52<33:14:13, 50.87s/it] {'loss': 0.8327, 'learning_rate': 3.771477493726535e-05, 'epoch': 0.54} 18%|█▊ | 513/2865 [3:31:52<33:14:13, 50.87s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 165.85 | bwd_microstep: 219.42 | bwd_inner_microstep: 219.17 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 181.26 | bwd_microstep: 222.44 | bwd_inner_microstep: 222.04 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.33 | bwd_microstep: 219.03 | bwd_inner_microstep: 218.72 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 163.59 | bwd_microstep: 217.73 | bwd_inner_microstep: 217.61 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 176.19 | bwd_microstep: 222.70 | bwd_inner_microstep: 222.54 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 161.84 | bwd_microstep: 219.83 | bwd_inner_microstep: 219.59 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 161.12 | bwd_microstep: 225.16 | bwd_inner_microstep: 224.85 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 160.48 | bwd_microstep: 232.09 | bwd_inner_microstep: 221.10 | bwd_allreduce_microstep: 10.83 | step_microstep: 21.01 [Rank 0] time (ms) | fwd: 1331.54 | bwd: 1778.52 | bwd_inner: 1766.12 | bwd_allreduce: 11.42 | step: 21.58 18%|█▊ | 514/2865 [3:31:55<23:53:57, 36.60s/it] {'loss': 0.8414, 'learning_rate': 3.770426864093995e-05, 'epoch': 0.54} 18%|█▊ | 514/2865 [3:31:55<23:53:57, 36.60s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 947 [Rank 0] time (ms) | fwd_microstep: 162.90 | bwd_microstep: 190.80 | bwd_inner_microstep: 190.67 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.39 | bwd_microstep: 191.69 | bwd_inner_microstep: 191.57 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 164.23 | bwd_microstep: 190.71 | bwd_inner_microstep: 190.46 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 160.93 | bwd_microstep: 194.10 | bwd_inner_microstep: 193.97 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 161.56 | bwd_microstep: 191.39 | bwd_inner_microstep: 191.02 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.92 | bwd_microstep: 191.04 | bwd_inner_microstep: 190.73 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 160.72 | bwd_microstep: 190.73 | bwd_inner_microstep: 190.44 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 147.60 | bwd_microstep: 200.70 | bwd_inner_microstep: 188.17 | bwd_allreduce_microstep: 12.37 | step_microstep: 21.00 [Rank 0] time (ms) | fwd: 1280.14 | bwd: 1541.27 | bwd_inner: 1527.42 | bwd_allreduce: 13.01 | step: 21.67 18%|█▊ | 515/2865 [3:31:58<17:18:27, 26.51s/it] {'loss': 0.8273, 'learning_rate': 3.769373971898893e-05, 'epoch': 0.54} 18%|█▊ | 515/2865 [3:31:58<17:18:27, 26.51s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.32 | bwd_microstep: 189.11 | bwd_inner_microstep: 188.99 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 163.63 | bwd_microstep: 190.50 | bwd_inner_microstep: 190.38 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.00 | bwd_microstep: 188.27 | bwd_inner_microstep: 187.94 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.29 | bwd_microstep: 191.92 | bwd_inner_microstep: 191.78 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 161.43 | bwd_microstep: 190.87 | bwd_inner_microstep: 190.59 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.09 | bwd_microstep: 189.88 | bwd_inner_microstep: 189.65 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.00 | bwd_microstep: 190.13 | bwd_inner_microstep: 189.84 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.89 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 147.06 | bwd_microstep: 217.29 | bwd_inner_microstep: 207.41 | bwd_allreduce_microstep: 9.77 | step_microstep: 21.12 [Rank 0] time (ms) | fwd: 1276.71 | bwd: 1548.06 | bwd_inner: 1536.95 | bwd_allreduce: 10.38 | step: 21.61 18%|█▊ | 516/2865 [3:32:01<12:41:48, 19.46s/it] {'loss': 0.8993, 'learning_rate': 3.768318818486801e-05, 'epoch': 0.54} 18%|█▊ | 516/2865 [3:32:01<12:41:48, 19.46s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 161.95 | bwd_microstep: 215.27 | bwd_inner_microstep: 215.13 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 160.19 | bwd_microstep: 191.99 | bwd_inner_microstep: 191.85 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 163.88 | bwd_microstep: 193.24 | bwd_inner_microstep: 193.12 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 160.10 | bwd_microstep: 189.79 | bwd_inner_microstep: 189.66 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.98 | bwd_microstep: 192.27 | bwd_inner_microstep: 191.90 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 160.70 | bwd_microstep: 187.91 | bwd_inner_microstep: 187.80 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.14 | bwd_microstep: 190.77 | bwd_inner_microstep: 190.64 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 162.60 | bwd_microstep: 218.02 | bwd_inner_microstep: 208.32 | bwd_allreduce_microstep: 9.61 | step_microstep: 20.98 [Rank 0] time (ms) | fwd: 1291.43 | bwd: 1579.37 | bwd_inner: 1568.62 | bwd_allreduce: 10.11 | step: 21.67 18%|█▊ | 517/2865 [3:32:04<9:28:36, 14.53s/it] {'loss': 1.0197, 'learning_rate': 3.7672614052061803e-05, 'epoch': 0.54} 18%|█▊ | 517/2865 [3:32:04<9:28:36, 14.53s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 162.00 | bwd_microstep: 191.35 | bwd_inner_microstep: 191.06 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 161.67 | bwd_microstep: 191.87 | bwd_inner_microstep: 191.75 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 160.90 | bwd_microstep: 192.68 | bwd_inner_microstep: 192.53 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.18 | bwd_microstep: 189.26 | bwd_inner_microstep: 189.14 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 146.99 | bwd_microstep: 174.83 | bwd_inner_microstep: 174.70 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.76 | bwd_microstep: 188.88 | bwd_inner_microstep: 188.76 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 160.03 | bwd_microstep: 187.86 | bwd_inner_microstep: 187.75 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.82 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 159.85 | bwd_microstep: 213.72 | bwd_inner_microstep: 203.96 | bwd_allreduce_microstep: 9.65 | step_microstep: 20.81 [Rank 0] time (ms) | fwd: 1273.29 | bwd: 1530.57 | bwd_inner: 1519.83 | bwd_allreduce: 10.12 | step: 21.40 18%|█▊ | 518/2865 [3:32:07<7:12:34, 11.06s/it] {'loss': 0.8864, 'learning_rate': 3.766201733408378e-05, 'epoch': 0.54} 18%|█▊ | 518/2865 [3:32:07<7:12:34, 11.06s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 160.80 | bwd_microstep: 212.51 | bwd_inner_microstep: 212.23 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.34 | bwd_microstep: 215.56 | bwd_inner_microstep: 215.28 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 161.53 | bwd_microstep: 212.90 | bwd_inner_microstep: 212.77 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.24 | bwd_microstep: 216.10 | bwd_inner_microstep: 215.81 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 162.26 | bwd_microstep: 214.70 | bwd_inner_microstep: 214.41 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 159.88 | bwd_microstep: 213.69 | bwd_inner_microstep: 213.39 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.42 | bwd_microstep: 215.99 | bwd_inner_microstep: 215.64 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 160.24 | bwd_microstep: 249.31 | bwd_inner_microstep: 239.52 | bwd_allreduce_microstep: 9.64 | step_microstep: 20.82 [Rank 0] time (ms) | fwd: 1286.61 | bwd: 1750.85 | bwd_inner: 1739.56 | bwd_allreduce: 10.34 | step: 21.32 18%|█▊ | 519/2865 [3:32:10<5:40:07, 8.70s/it] {'loss': 0.97, 'learning_rate': 3.7651398044476295e-05, 'epoch': 0.54} 18%|█▊ | 519/2865 [3:32:10<5:40:07, 8.70s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 952 [Rank 0] time (ms) | fwd_microstep: 162.49 | bwd_microstep: 216.28 | bwd_inner_microstep: 215.99 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 160.89 | bwd_microstep: 220.53 | bwd_inner_microstep: 220.18 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 161.68 | bwd_microstep: 216.88 | bwd_inner_microstep: 216.61 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 161.31 | bwd_microstep: 215.44 | bwd_inner_microstep: 215.16 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.56 | bwd_microstep: 216.81 | bwd_inner_microstep: 216.51 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 160.57 | bwd_microstep: 219.12 | bwd_inner_microstep: 218.74 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.92 | bwd_microstep: 214.86 | bwd_inner_microstep: 214.54 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.42 | bwd_microstep: 247.89 | bwd_inner_microstep: 238.05 | bwd_allreduce_microstep: 9.68 | step_microstep: 21.13 [Rank 0] time (ms) | fwd: 1288.74 | bwd: 1767.95 | bwd_inner: 1756.32 | bwd_allreduce: 10.54 | step: 21.86 18%|█▊ | 520/2865 [3:32:13<4:35:40, 7.05s/it] {'loss': 0.8077, 'learning_rate': 3.7640756196810545e-05, 'epoch': 0.54} 18%|█▊ | 520/2865 [3:32:13<4:35:40, 7.05s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 160.95 | bwd_microstep: 216.42 | bwd_inner_microstep: 216.05 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.62 | bwd_microstep: 218.01 | bwd_inner_microstep: 217.72 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 163.53 | bwd_microstep: 260.52 | bwd_inner_microstep: 260.24 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 170.87 | bwd_microstep: 259.40 | bwd_inner_microstep: 259.25 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 171.74 | bwd_microstep: 260.98 | bwd_inner_microstep: 260.71 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 172.29 | bwd_microstep: 262.26 | bwd_inner_microstep: 261.98 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 170.85 | bwd_microstep: 248.26 | bwd_inner_microstep: 247.98 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 170.60 | bwd_microstep: 306.38 | bwd_inner_microstep: 295.50 | bwd_allreduce_microstep: 10.70 | step_microstep: 21.31 [Rank 0] time (ms) | fwd: 1341.33 | bwd: 2032.35 | bwd_inner: 2019.96 | bwd_allreduce: 11.39 | step: 22.07 18%|█▊ | 521/2865 [3:32:17<3:54:32, 6.00s/it] {'loss': 0.9085, 'learning_rate': 3.763009180468655e-05, 'epoch': 0.55} 18%|█▊ | 521/2865 [3:32:17<3:54:32, 6.00s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 161.21 | bwd_microstep: 216.37 | bwd_inner_microstep: 216.23 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 161.35 | bwd_microstep: 217.23 | bwd_inner_microstep: 216.92 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.66 | bwd_microstep: 216.91 | bwd_inner_microstep: 216.77 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 160.99 | bwd_microstep: 215.59 | bwd_inner_microstep: 215.46 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.35 | bwd_microstep: 213.57 | bwd_inner_microstep: 213.20 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.43 | bwd_microstep: 187.90 | bwd_inner_microstep: 187.80 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 159.89 | bwd_microstep: 187.35 | bwd_inner_microstep: 187.22 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.48 | bwd_microstep: 211.92 | bwd_inner_microstep: 202.46 | bwd_allreduce_microstep: 9.38 | step_microstep: 20.81 [Rank 0] time (ms) | fwd: 1285.23 | bwd: 1666.94 | bwd_inner: 1656.32 | bwd_allreduce: 9.95 | step: 21.41 18%|█▊ | 522/2865 [3:32:20<3:21:13, 5.15s/it] {'loss': 0.8263, 'learning_rate': 3.761940488173315e-05, 'epoch': 0.55} 18%|█▊ | 522/2865 [3:32:20<3:21:13, 5.15s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 947 [Rank 0] time (ms) | fwd_microstep: 163.50 | bwd_microstep: 190.64 | bwd_inner_microstep: 190.51 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 160.94 | bwd_microstep: 189.40 | bwd_inner_microstep: 189.11 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.75 | bwd_microstep: 190.12 | bwd_inner_microstep: 189.73 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.43 | bwd_microstep: 189.97 | bwd_inner_microstep: 189.86 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 170.55 | bwd_microstep: 224.81 | bwd_inner_microstep: 224.44 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 164.79 | bwd_microstep: 217.22 | bwd_inner_microstep: 217.09 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 159.93 | bwd_microstep: 215.64 | bwd_inner_microstep: 215.32 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.37 | bwd_microstep: 247.22 | bwd_inner_microstep: 237.45 | bwd_allreduce_microstep: 9.62 | step_microstep: 21.01 [Rank 0] time (ms) | fwd: 1302.15 | bwd: 1665.12 | bwd_inner: 1653.90 | bwd_allreduce: 10.31 | step: 21.62 18%|█▊ | 523/2865 [3:32:23<2:57:25, 4.55s/it] {'loss': 1.1356, 'learning_rate': 3.7608695441607957e-05, 'epoch': 0.55} 18%|█▊ | 523/2865 [3:32:23<2:57:25, 4.55s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 161.39 | bwd_microstep: 222.74 | bwd_inner_microstep: 222.58 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 177.80 | bwd_microstep: 221.95 | bwd_inner_microstep: 221.58 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 162.94 | bwd_microstep: 222.01 | bwd_inner_microstep: 221.75 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.96 | bwd_microstep: 228.53 | bwd_inner_microstep: 228.16 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 162.55 | bwd_microstep: 221.28 | bwd_inner_microstep: 220.98 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 165.18 | bwd_microstep: 222.73 | bwd_inner_microstep: 222.54 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 165.37 | bwd_microstep: 222.57 | bwd_inner_microstep: 222.45 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.80 | bwd_microstep: 258.70 | bwd_inner_microstep: 248.11 | bwd_allreduce_microstep: 10.52 | step_microstep: 21.14 [Rank 0] time (ms) | fwd: 1319.87 | bwd: 1820.63 | bwd_inner: 1808.56 | bwd_allreduce: 11.20 | step: 21.89 18%|█▊ | 524/2865 [3:32:26<2:42:57, 4.18s/it] {'loss': 1.0908, 'learning_rate': 3.759796349799738e-05, 'epoch': 0.55} 18%|█▊ | 524/2865 [3:32:26<2:42:57, 4.18s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 161.05 | bwd_microstep: 195.20 | bwd_inner_microstep: 194.84 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.25 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 162.14 | bwd_microstep: 191.72 | bwd_inner_microstep: 191.51 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.52 | bwd_microstep: 191.59 | bwd_inner_microstep: 191.26 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.92 | bwd_microstep: 199.50 | bwd_inner_microstep: 199.10 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 161.63 | bwd_microstep: 199.68 | bwd_inner_microstep: 199.32 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 161.96 | bwd_microstep: 194.29 | bwd_inner_microstep: 193.93 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 164.95 | bwd_microstep: 224.68 | bwd_inner_microstep: 224.44 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 147.31 | bwd_microstep: 202.87 | bwd_inner_microstep: 191.45 | bwd_allreduce_microstep: 11.26 | step_microstep: 21.11 [Rank 0] time (ms) | fwd: 1282.35 | bwd: 1599.74 | bwd_inner: 1586.41 | bwd_allreduce: 12.14 | step: 22.21 18%|█▊ | 525/2865 [3:32:30<2:29:53, 3.84s/it] {'loss': 0.7691, 'learning_rate': 3.75872090646166e-05, 'epoch': 0.55} 18%|█▊ | 525/2865 [3:32:30<2:29:53, 3.84s/it][INFO|trainer.py:3242] 2026-05-14 23:53:12,744 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 23:53:12,745 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 23:53:12,745 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-525 [INFO|configuration_utils.py:473] 2026-05-14 23:58:18,474 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-525/config.json [INFO|configuration_utils.py:594] 2026-05-14 23:58:18,482 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-525/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-14 23:58:23,407 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-525/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-14 23:58:23,415 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-525/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-14 23:58:23,421 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-525/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-14 23:58:23,422 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-525/added_tokens.json [INFO|trainer.py:3028] 2026-05-14 23:58:33,099 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-450] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 302916.97 | bwd_microstep: 219.80 | bwd_inner_microstep: 219.63 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 146.29 | bwd_microstep: 192.11 | bwd_inner_microstep: 191.73 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.36 | bwd_microstep: 218.73 | bwd_inner_microstep: 218.60 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.79 | bwd_microstep: 217.65 | bwd_inner_microstep: 217.49 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 160.52 | bwd_microstep: 219.33 | bwd_inner_microstep: 218.96 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 161.88 | bwd_microstep: 222.17 | bwd_inner_microstep: 222.03 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.81 | bwd_microstep: 217.94 | bwd_inner_microstep: 217.66 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.92 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.57 | bwd_microstep: 249.39 | bwd_inner_microstep: 239.25 | bwd_allreduce_microstep: 9.99 | step_microstep: 21.26 [Rank 0] time (ms) | fwd: 304019.88 | bwd: 1757.25 | bwd_inner: 1745.74 | bwd_allreduce: 10.65 | step: 22.01 18%|█▊ | 526/2865 [3:37:54<64:55:46, 99.93s/it] {'loss': 0.8672, 'learning_rate': 3.757643215520951e-05, 'epoch': 0.55} 18%|█▊ | 526/2865 [3:37:54<64:55:46, 99.93s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 162.78 | bwd_microstep: 218.12 | bwd_inner_microstep: 217.76 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.53 | bwd_microstep: 218.47 | bwd_inner_microstep: 218.34 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 178.33 | bwd_microstep: 216.13 | bwd_inner_microstep: 216.00 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.25 | bwd_microstep: 215.87 | bwd_inner_microstep: 215.59 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 163.31 | bwd_microstep: 216.92 | bwd_inner_microstep: 216.67 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 163.00 | bwd_microstep: 219.64 | bwd_inner_microstep: 219.36 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 164.01 | bwd_microstep: 223.53 | bwd_inner_microstep: 223.07 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 161.99 | bwd_microstep: 249.92 | bwd_inner_microstep: 239.61 | bwd_allreduce_microstep: 10.15 | step_microstep: 21.08 [Rank 0] time (ms) | fwd: 1316.11 | bwd: 1778.76 | bwd_inner: 1766.83 | bwd_allreduce: 10.91 | step: 22.02 18%|█▊ | 527/2865 [3:37:57<46:03:51, 70.93s/it] {'loss': 0.8334, 'learning_rate': 3.756563278354874e-05, 'epoch': 0.55} 18%|█▊ | 527/2865 [3:37:57<46:03:51, 70.93s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 162.21 | bwd_microstep: 217.04 | bwd_inner_microstep: 216.91 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.41 | bwd_microstep: 215.88 | bwd_inner_microstep: 215.58 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 161.48 | bwd_microstep: 211.25 | bwd_inner_microstep: 210.92 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.46 | bwd_microstep: 189.83 | bwd_inner_microstep: 189.71 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.50 | bwd_microstep: 185.84 | bwd_inner_microstep: 185.56 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 161.88 | bwd_microstep: 187.54 | bwd_inner_microstep: 187.48 | bwd_allreduce_microstep: 0.02 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.19 | bwd_microstep: 187.66 | bwd_inner_microstep: 187.55 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.77 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 155.90 | bwd_microstep: 214.20 | bwd_inner_microstep: 206.14 | bwd_allreduce_microstep: 7.99 | step_microstep: 20.76 [Rank 0] time (ms) | fwd: 1289.93 | bwd: 1609.30 | bwd_inner: 1600.09 | bwd_allreduce: 8.51 | step: 21.27 18%|█▊ | 528/2865 [3:38:00<32:49:31, 50.57s/it] {'loss': 0.948, 'learning_rate': 3.7554810963435626e-05, 'epoch': 0.55} 18%|█▊ | 528/2865 [3:38:00<32:49:31, 50.57s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 156.57 | bwd_microstep: 194.94 | bwd_inner_microstep: 194.82 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 156.47 | bwd_microstep: 188.23 | bwd_inner_microstep: 188.13 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 157.54 | bwd_microstep: 190.31 | bwd_inner_microstep: 190.13 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 156.31 | bwd_microstep: 186.97 | bwd_inner_microstep: 186.68 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 155.69 | bwd_microstep: 188.27 | bwd_inner_microstep: 188.07 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 155.70 | bwd_microstep: 188.49 | bwd_inner_microstep: 188.20 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 157.35 | bwd_microstep: 189.14 | bwd_inner_microstep: 189.07 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.76 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 156.69 | bwd_microstep: 211.61 | bwd_inner_microstep: 203.67 | bwd_allreduce_microstep: 7.88 | step_microstep: 20.54 [Rank 0] time (ms) | fwd: 1252.23 | bwd: 1538.04 | bwd_inner: 1529.10 | bwd_allreduce: 8.37 | step: 20.97 18%|█▊ | 529/2865 [3:38:03<23:32:21, 36.28s/it] {'loss': 1.0321, 'learning_rate': 3.754396670870018e-05, 'epoch': 0.55} 18%|█▊ | 529/2865 [3:38:03<23:32:21, 36.28s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 155.63 | bwd_microstep: 187.55 | bwd_inner_microstep: 187.44 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 156.76 | bwd_microstep: 190.73 | bwd_inner_microstep: 190.59 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 163.65 | bwd_microstep: 189.09 | bwd_inner_microstep: 188.66 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 158.98 | bwd_microstep: 190.94 | bwd_inner_microstep: 190.58 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.27 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.56 | bwd_microstep: 189.56 | bwd_inner_microstep: 189.28 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 157.63 | bwd_microstep: 190.61 | bwd_inner_microstep: 190.53 | bwd_allreduce_microstep: 0.04 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 162.68 | bwd_microstep: 188.50 | bwd_inner_microstep: 188.12 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.79 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 155.71 | bwd_microstep: 218.38 | bwd_inner_microstep: 207.68 | bwd_allreduce_microstep: 10.53 | step_microstep: 20.64 [Rank 0] time (ms) | fwd: 1271.45 | bwd: 1545.53 | bwd_inner: 1533.29 | bwd_allreduce: 11.26 | step: 21.60 18%|█▊ | 530/2865 [3:38:06<17:03:01, 26.29s/it] {'loss': 0.8359, 'learning_rate': 3.753310003320111e-05, 'epoch': 0.55} 18%|█▊ | 530/2865 [3:38:06<17:03:01, 26.29s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 157.87 | bwd_microstep: 189.60 | bwd_inner_microstep: 189.23 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.25 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 160.32 | bwd_microstep: 189.68 | bwd_inner_microstep: 189.46 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 158.56 | bwd_microstep: 189.45 | bwd_inner_microstep: 189.33 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 909 [Rank 0] time (ms) | fwd_microstep: 146.42 | bwd_microstep: 172.86 | bwd_inner_microstep: 172.55 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 158.38 | bwd_microstep: 188.05 | bwd_inner_microstep: 187.93 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 156.42 | bwd_microstep: 188.24 | bwd_inner_microstep: 187.94 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 157.71 | bwd_microstep: 188.42 | bwd_inner_microstep: 188.37 | bwd_allreduce_microstep: 0.02 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.80 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 144.06 | bwd_microstep: 191.40 | bwd_inner_microstep: 183.24 | bwd_allreduce_microstep: 8.02 | step_microstep: 20.85 [Rank 0] time (ms) | fwd: 1239.61 | bwd: 1497.79 | bwd_inner: 1488.49 | bwd_allreduce: 8.55 | step: 21.53 19%|█▊ | 531/2865 [3:38:09<12:29:40, 19.27s/it] {'loss': 0.9983, 'learning_rate': 3.752221095082576e-05, 'epoch': 0.56} 19%|█▊ | 531/2865 [3:38:09<12:29:40, 19.27s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 155.97 | bwd_microstep: 186.43 | bwd_inner_microstep: 186.21 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 156.34 | bwd_microstep: 190.41 | bwd_inner_microstep: 190.23 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 158.47 | bwd_microstep: 191.37 | bwd_inner_microstep: 190.86 | bwd_allreduce_microstep: 0.22 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 158.39 | bwd_microstep: 189.25 | bwd_inner_microstep: 188.96 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 155.66 | bwd_microstep: 187.95 | bwd_inner_microstep: 187.70 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 156.75 | bwd_microstep: 191.44 | bwd_inner_microstep: 191.17 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 156.00 | bwd_microstep: 186.26 | bwd_inner_microstep: 186.15 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 155.96 | bwd_microstep: 217.24 | bwd_inner_microstep: 207.35 | bwd_allreduce_microstep: 9.74 | step_microstep: 21.19 [Rank 0] time (ms) | fwd: 1253.44 | bwd: 1540.46 | bwd_inner: 1529.08 | bwd_allreduce: 10.39 | step: 21.82 19%|█▊ | 532/2865 [3:38:12<9:18:49, 14.37s/it] {'loss': 0.8565, 'learning_rate': 3.7511299475490115e-05, 'epoch': 0.56} 19%|█▊ | 532/2865 [3:38:12<9:18:49, 14.37s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 160.67 | bwd_microstep: 189.30 | bwd_inner_microstep: 189.00 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 160.33 | bwd_microstep: 189.61 | bwd_inner_microstep: 189.25 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 161.26 | bwd_microstep: 190.10 | bwd_inner_microstep: 189.76 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 146.66 | bwd_microstep: 173.83 | bwd_inner_microstep: 173.66 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.36 | bwd_microstep: 188.46 | bwd_inner_microstep: 188.30 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.42 | bwd_microstep: 186.03 | bwd_inner_microstep: 185.74 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 146.93 | bwd_microstep: 172.97 | bwd_inner_microstep: 172.64 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 163.35 | bwd_microstep: 214.22 | bwd_inner_microstep: 204.50 | bwd_allreduce_microstep: 9.63 | step_microstep: 20.90 [Rank 0] time (ms) | fwd: 1259.86 | bwd: 1504.63 | bwd_inner: 1493.30 | bwd_allreduce: 10.49 | step: 21.53 19%|█▊ | 533/2865 [3:38:15<7:05:03, 10.94s/it] {'loss': 0.7781, 'learning_rate': 3.750036562113877e-05, 'epoch': 0.56} 19%|█▊ | 533/2865 [3:38:15<7:05:03, 10.94s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 161.14 | bwd_microstep: 187.12 | bwd_inner_microstep: 186.99 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 158.12 | bwd_microstep: 191.33 | bwd_inner_microstep: 191.03 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 164.35 | bwd_microstep: 190.12 | bwd_inner_microstep: 189.85 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 159.16 | bwd_microstep: 189.01 | bwd_inner_microstep: 188.73 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 164.25 | bwd_microstep: 188.79 | bwd_inner_microstep: 188.45 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 156.87 | bwd_microstep: 193.49 | bwd_inner_microstep: 193.33 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 160.22 | bwd_microstep: 189.40 | bwd_inner_microstep: 189.27 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 162.06 | bwd_microstep: 219.25 | bwd_inner_microstep: 206.69 | bwd_allreduce_microstep: 12.39 | step_microstep: 22.47 [Rank 0] time (ms) | fwd: 1286.07 | bwd: 1548.57 | bwd_inner: 1534.80 | bwd_allreduce: 12.99 | step: 23.02 19%|█▊ | 534/2865 [3:38:18<5:32:19, 8.55s/it] {'loss': 0.7394, 'learning_rate': 3.748940940174493e-05, 'epoch': 0.56} 19%|█▊ | 534/2865 [3:38:18<5:32:19, 8.55s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 162.44 | bwd_microstep: 191.30 | bwd_inner_microstep: 191.17 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 163.65 | bwd_microstep: 192.83 | bwd_inner_microstep: 192.57 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 163.04 | bwd_microstep: 193.00 | bwd_inner_microstep: 192.64 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 162.53 | bwd_microstep: 236.64 | bwd_inner_microstep: 236.34 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 163.49 | bwd_microstep: 220.54 | bwd_inner_microstep: 220.38 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 161.71 | bwd_microstep: 219.59 | bwd_inner_microstep: 219.23 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 162.36 | bwd_microstep: 227.45 | bwd_inner_microstep: 227.16 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.10 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.07 | bwd_microstep: 266.72 | bwd_inner_microstep: 250.13 | bwd_allreduce_microstep: 16.47 | step_microstep: 28.61 [Rank 0] time (ms) | fwd: 1301.18 | bwd: 1748.23 | bwd_inner: 1730.06 | bwd_allreduce: 17.20 | step: 29.35 19%|█▊ | 535/2865 [3:38:21<4:30:01, 6.95s/it] {'loss': 0.7705, 'learning_rate': 3.747843083131037e-05, 'epoch': 0.56} 19%|█▊ | 535/2865 [3:38:21<4:30:01, 6.95s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 164.70 | bwd_microstep: 234.50 | bwd_inner_microstep: 234.22 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 196.00 | bwd_microstep: 264.85 | bwd_inner_microstep: 264.57 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 188.03 | bwd_microstep: 270.65 | bwd_inner_microstep: 270.47 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 175.57 | bwd_microstep: 264.41 | bwd_inner_microstep: 264.18 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 177.18 | bwd_microstep: 264.49 | bwd_inner_microstep: 264.27 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 213.49 | bwd_microstep: 264.66 | bwd_inner_microstep: 264.47 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 180.80 | bwd_microstep: 271.43 | bwd_inner_microstep: 271.13 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.11 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 159.48 | bwd_microstep: 273.75 | bwd_inner_microstep: 262.44 | bwd_allreduce_microstep: 11.17 | step_microstep: 21.91 [Rank 0] time (ms) | fwd: 1455.14 | bwd: 2108.85 | bwd_inner: 2096.16 | bwd_allreduce: 11.79 | step: 22.70 19%|█▊ | 536/2865 [3:38:25<3:52:42, 6.00s/it] {'loss': 0.7674, 'learning_rate': 3.746742992386544e-05, 'epoch': 0.56} 19%|█▊ | 536/2865 [3:38:25<3:52:42, 6.00s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 941 [Rank 0] time (ms) | fwd_microstep: 172.91 | bwd_microstep: 271.40 | bwd_inner_microstep: 271.20 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 178.41 | bwd_microstep: 271.27 | bwd_inner_microstep: 271.06 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 179.16 | bwd_microstep: 265.79 | bwd_inner_microstep: 265.52 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 179.62 | bwd_microstep: 268.69 | bwd_inner_microstep: 268.47 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 180.85 | bwd_microstep: 268.36 | bwd_inner_microstep: 268.20 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 181.13 | bwd_microstep: 268.05 | bwd_inner_microstep: 267.87 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 180.34 | bwd_microstep: 270.92 | bwd_inner_microstep: 270.75 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.43 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 181.80 | bwd_microstep: 306.64 | bwd_inner_microstep: 295.84 | bwd_allreduce_microstep: 10.69 | step_microstep: 23.74 [Rank 0] time (ms) | fwd: 1434.11 | bwd: 2191.21 | bwd_inner: 2179.21 | bwd_allreduce: 11.33 | step: 24.46 19%|█▊ | 537/2865 [3:38:28<3:27:09, 5.34s/it] {'loss': 0.8039, 'learning_rate': 3.7456406693469044e-05, 'epoch': 0.56} 19%|█▊ | 537/2865 [3:38:28<3:27:09, 5.34s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 175.51 | bwd_microstep: 258.57 | bwd_inner_microstep: 258.43 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 163.70 | bwd_microstep: 238.12 | bwd_inner_microstep: 237.86 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 181.96 | bwd_microstep: 261.33 | bwd_inner_microstep: 260.96 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 178.65 | bwd_microstep: 265.66 | bwd_inner_microstep: 265.38 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 174.81 | bwd_microstep: 258.93 | bwd_inner_microstep: 258.53 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 173.17 | bwd_microstep: 262.36 | bwd_inner_microstep: 262.21 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 169.64 | bwd_microstep: 260.30 | bwd_inner_microstep: 260.11 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.90 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 175.34 | bwd_microstep: 296.79 | bwd_inner_microstep: 286.22 | bwd_allreduce_microstep: 10.47 | step_microstep: 21.57 [Rank 0] time (ms) | fwd: 1392.67 | bwd: 2102.20 | bwd_inner: 2090.08 | bwd_allreduce: 11.20 | step: 22.40 19%|█▉ | 538/2865 [3:38:32<3:07:43, 4.84s/it] {'loss': 0.9742, 'learning_rate': 3.744536115420859e-05, 'epoch': 0.56} 19%|█▉ | 538/2865 [3:38:32<3:07:43, 4.84s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 173.96 | bwd_microstep: 255.76 | bwd_inner_microstep: 255.59 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 166.81 | bwd_microstep: 267.70 | bwd_inner_microstep: 267.42 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 177.42 | bwd_microstep: 266.70 | bwd_inner_microstep: 266.55 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 172.32 | bwd_microstep: 271.50 | bwd_inner_microstep: 271.27 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 176.65 | bwd_microstep: 266.28 | bwd_inner_microstep: 266.12 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 168.06 | bwd_microstep: 259.30 | bwd_inner_microstep: 259.16 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 174.89 | bwd_microstep: 261.01 | bwd_inner_microstep: 260.50 | bwd_allreduce_microstep: 0.20 | step_microstep: 0.27 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.07 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 169.36 | bwd_microstep: 298.85 | bwd_inner_microstep: 288.08 | bwd_allreduce_microstep: 10.63 | step_microstep: 23.80 [Rank 0] time (ms) | fwd: 1379.33 | bwd: 2147.24 | bwd_inner: 2135.08 | bwd_allreduce: 11.32 | step: 24.70 19%|█▉ | 539/2865 [3:38:36<2:54:47, 4.51s/it] {'loss': 0.8173, 'learning_rate': 3.743429332020001e-05, 'epoch': 0.56} 19%|█▉ | 539/2865 [3:38:36<2:54:47, 4.51s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 170.82 | bwd_microstep: 272.51 | bwd_inner_microstep: 272.34 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 173.23 | bwd_microstep: 274.06 | bwd_inner_microstep: 273.90 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 176.67 | bwd_microstep: 270.30 | bwd_inner_microstep: 270.15 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 166.06 | bwd_microstep: 267.02 | bwd_inner_microstep: 266.85 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 173.37 | bwd_microstep: 265.20 | bwd_inner_microstep: 264.80 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 176.49 | bwd_microstep: 264.62 | bwd_inner_microstep: 264.14 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 169.59 | bwd_microstep: 260.86 | bwd_inner_microstep: 260.71 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.91 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 173.15 | bwd_microstep: 309.02 | bwd_inner_microstep: 298.30 | bwd_allreduce_microstep: 10.62 | step_microstep: 21.41 [Rank 0] time (ms) | fwd: 1379.24 | bwd: 2183.75 | bwd_inner: 2171.55 | bwd_allreduce: 11.36 | step: 22.32 19%|█▉ | 540/2865 [3:38:40<2:45:54, 4.28s/it] {'loss': 1.0856, 'learning_rate': 3.7423203205587737e-05, 'epoch': 0.56} 19%|█▉ | 540/2865 [3:38:40<2:45:54, 4.28s/it][INFO|trainer.py:3242] 2026-05-14 23:59:22,812 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-14 23:59:22,812 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-14 23:59:22,812 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-540 [INFO|configuration_utils.py:473] 2026-05-15 00:04:28,820 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-540/config.json [INFO|configuration_utils.py:594] 2026-05-15 00:04:28,827 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-540/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-15 00:04:34,474 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-540/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-15 00:04:34,482 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-540/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-15 00:04:34,488 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-540/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-15 00:04:34,489 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-540/added_tokens.json [INFO|trainer.py:3028] 2026-05-15 00:04:43,860 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-465] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 302682.56 | bwd_microstep: 248.65 | bwd_inner_microstep: 248.36 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 160.51 | bwd_microstep: 215.29 | bwd_inner_microstep: 215.11 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.03 | bwd_microstep: 224.41 | bwd_inner_microstep: 224.12 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 160.59 | bwd_microstep: 226.17 | bwd_inner_microstep: 225.63 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.52 | bwd_microstep: 251.34 | bwd_inner_microstep: 250.90 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.99 | bwd_microstep: 232.63 | bwd_inner_microstep: 232.31 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 164.24 | bwd_microstep: 220.42 | bwd_inner_microstep: 220.16 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.88 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.44 | bwd_microstep: 253.30 | bwd_inner_microstep: 242.81 | bwd_allreduce_microstep: 10.35 | step_microstep: 21.26 [Rank 0] time (ms) | fwd: 303800.44 | bwd: 1872.35 | bwd_inner: 1860.06 | bwd_allreduce: 11.18 | step: 22.13 19%|█▉ | 541/2865 [3:44:05<64:52:14, 100.49s/it] {'loss': 0.9147, 'learning_rate': 3.741209082454466e-05, 'epoch': 0.57} 19%|█▉ | 541/2865 [3:44:05<64:52:14, 100.49s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 161.55 | bwd_microstep: 218.64 | bwd_inner_microstep: 218.28 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 163.92 | bwd_microstep: 227.43 | bwd_inner_microstep: 227.30 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 167.57 | bwd_microstep: 234.16 | bwd_inner_microstep: 233.84 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 160.65 | bwd_microstep: 220.17 | bwd_inner_microstep: 219.90 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 167.11 | bwd_microstep: 226.89 | bwd_inner_microstep: 226.46 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 162.09 | bwd_microstep: 228.75 | bwd_inner_microstep: 228.61 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 164.88 | bwd_microstep: 229.14 | bwd_inner_microstep: 228.87 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.48 | bwd_microstep: 216.65 | bwd_inner_microstep: 206.79 | bwd_allreduce_microstep: 9.70 | step_microstep: 21.62 [Rank 0] time (ms) | fwd: 1311.10 | bwd: 1801.96 | bwd_inner: 1790.58 | bwd_allreduce: 10.47 | step: 22.42 19%|█▉ | 542/2865 [3:44:08<46:01:45, 71.33s/it] {'loss': 0.9475, 'learning_rate': 3.740095619127213e-05, 'epoch': 0.57} 19%|█▉ | 542/2865 [3:44:08<46:01:45, 71.33s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 160.12 | bwd_microstep: 190.05 | bwd_inner_microstep: 189.76 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 160.53 | bwd_microstep: 189.64 | bwd_inner_microstep: 189.54 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 164.42 | bwd_microstep: 191.50 | bwd_inner_microstep: 191.40 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.63 | bwd_microstep: 190.11 | bwd_inner_microstep: 189.79 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 163.20 | bwd_microstep: 189.10 | bwd_inner_microstep: 188.99 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 163.84 | bwd_microstep: 190.42 | bwd_inner_microstep: 190.31 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 163.57 | bwd_microstep: 187.89 | bwd_inner_microstep: 187.69 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.59 | bwd_microstep: 214.49 | bwd_inner_microstep: 203.90 | bwd_allreduce_microstep: 10.39 | step_microstep: 20.82 [Rank 0] time (ms) | fwd: 1297.79 | bwd: 1543.29 | bwd_inner: 1531.72 | bwd_allreduce: 10.84 | step: 21.33 19%|█▉ | 543/2865 [3:44:11<32:47:19, 50.84s/it] {'loss': 0.9183, 'learning_rate': 3.738979931999994e-05, 'epoch': 0.57} 19%|█▉ | 543/2865 [3:44:11<32:47:19, 50.84s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 165.52 | bwd_microstep: 188.98 | bwd_inner_microstep: 188.85 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 160.39 | bwd_microstep: 189.32 | bwd_inner_microstep: 189.02 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.86 | bwd_microstep: 219.35 | bwd_inner_microstep: 218.99 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.30 | bwd_microstep: 218.12 | bwd_inner_microstep: 217.75 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.53 | bwd_microstep: 219.25 | bwd_inner_microstep: 218.88 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.32 | bwd_microstep: 223.21 | bwd_inner_microstep: 222.81 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 161.12 | bwd_microstep: 219.56 | bwd_inner_microstep: 219.28 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.07 | bwd_microstep: 249.24 | bwd_inner_microstep: 239.50 | bwd_allreduce_microstep: 9.57 | step_microstep: 21.06 [Rank 0] time (ms) | fwd: 1288.99 | bwd: 1727.21 | bwd_inner: 1715.61 | bwd_allreduce: 10.47 | step: 21.96 19%|█▉ | 544/2865 [3:44:14<23:33:32, 36.54s/it] {'loss': 0.8878, 'learning_rate': 3.737862022498631e-05, 'epoch': 0.57} 19%|█▉ | 544/2865 [3:44:14<23:33:32, 36.54s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 161.49 | bwd_microstep: 217.21 | bwd_inner_microstep: 216.94 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 942 [Rank 0] time (ms) | fwd_microstep: 162.04 | bwd_microstep: 228.34 | bwd_inner_microstep: 228.22 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 165.26 | bwd_microstep: 222.29 | bwd_inner_microstep: 222.02 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 162.18 | bwd_microstep: 222.06 | bwd_inner_microstep: 221.93 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.11 | bwd_microstep: 225.82 | bwd_inner_microstep: 225.65 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 161.14 | bwd_microstep: 233.14 | bwd_inner_microstep: 233.01 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.39 | bwd_microstep: 221.20 | bwd_inner_microstep: 220.88 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.83 | bwd_microstep: 258.61 | bwd_inner_microstep: 248.28 | bwd_allreduce_microstep: 10.22 | step_microstep: 20.78 [Rank 0] time (ms) | fwd: 1296.33 | bwd: 1828.80 | bwd_inner: 1817.27 | bwd_allreduce: 10.79 | step: 21.46 19%|█▉ | 545/2865 [3:44:17<17:07:09, 26.56s/it] {'loss': 0.8737, 'learning_rate': 3.736741892051784e-05, 'epoch': 0.57} 19%|█▉ | 545/2865 [3:44:17<17:07:09, 26.56s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 161.79 | bwd_microstep: 221.31 | bwd_inner_microstep: 221.04 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 161.51 | bwd_microstep: 195.79 | bwd_inner_microstep: 195.63 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 161.09 | bwd_microstep: 190.35 | bwd_inner_microstep: 190.19 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.74 | bwd_microstep: 201.21 | bwd_inner_microstep: 200.97 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.43 | bwd_microstep: 194.14 | bwd_inner_microstep: 193.86 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.33 | bwd_microstep: 191.54 | bwd_inner_microstep: 191.10 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.53 | bwd_microstep: 191.23 | bwd_inner_microstep: 191.09 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 160.38 | bwd_microstep: 215.31 | bwd_inner_microstep: 205.58 | bwd_allreduce_microstep: 9.59 | step_microstep: 20.92 [Rank 0] time (ms) | fwd: 1289.71 | bwd: 1601.02 | bwd_inner: 1589.91 | bwd_allreduce: 10.21 | step: 21.60 19%|█▉ | 546/2865 [3:44:20<12:34:02, 19.51s/it] {'loss': 0.9091, 'learning_rate': 3.735619542090953e-05, 'epoch': 0.57} 19%|█▉ | 546/2865 [3:44:20<12:34:02, 19.51s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 161.06 | bwd_microstep: 190.75 | bwd_inner_microstep: 190.63 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 162.00 | bwd_microstep: 216.73 | bwd_inner_microstep: 216.43 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 166.53 | bwd_microstep: 216.75 | bwd_inner_microstep: 216.46 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 160.83 | bwd_microstep: 221.09 | bwd_inner_microstep: 220.80 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.47 | bwd_microstep: 214.52 | bwd_inner_microstep: 214.15 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 160.23 | bwd_microstep: 215.32 | bwd_inner_microstep: 215.19 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 159.93 | bwd_microstep: 216.75 | bwd_inner_microstep: 216.45 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 159.95 | bwd_microstep: 249.08 | bwd_inner_microstep: 238.23 | bwd_allreduce_microstep: 10.67 | step_microstep: 20.85 [Rank 0] time (ms) | fwd: 1290.88 | bwd: 1741.08 | bwd_inner: 1728.79 | bwd_allreduce: 11.39 | step: 21.47 19%|█▉ | 547/2865 [3:44:24<9:24:36, 14.61s/it] {'loss': 0.8489, 'learning_rate': 3.734494974050474e-05, 'epoch': 0.57} 19%|█▉ | 547/2865 [3:44:24<9:24:36, 14.61s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 166.50 | bwd_microstep: 214.69 | bwd_inner_microstep: 214.41 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 160.57 | bwd_microstep: 226.30 | bwd_inner_microstep: 226.01 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.29 | bwd_microstep: 232.79 | bwd_inner_microstep: 232.56 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 160.36 | bwd_microstep: 223.03 | bwd_inner_microstep: 222.65 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 160.19 | bwd_microstep: 217.54 | bwd_inner_microstep: 217.42 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.44 | bwd_microstep: 222.42 | bwd_inner_microstep: 222.11 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 163.62 | bwd_microstep: 218.85 | bwd_inner_microstep: 218.53 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.25 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.84 | bwd_microstep: 252.56 | bwd_inner_microstep: 241.15 | bwd_allreduce_microstep: 11.25 | step_microstep: 21.58 [Rank 0] time (ms) | fwd: 1292.70 | bwd: 1808.32 | bwd_inner: 1795.30 | bwd_allreduce: 12.06 | step: 22.13 19%|█▉ | 548/2865 [3:44:27<7:12:50, 11.21s/it] {'loss': 0.9828, 'learning_rate': 3.733368189367518e-05, 'epoch': 0.57} 19%|█▉ | 548/2865 [3:44:27<7:12:50, 11.21s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 163.17 | bwd_microstep: 217.10 | bwd_inner_microstep: 216.81 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 162.15 | bwd_microstep: 221.35 | bwd_inner_microstep: 220.97 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 161.63 | bwd_microstep: 223.42 | bwd_inner_microstep: 223.03 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 163.45 | bwd_microstep: 209.51 | bwd_inner_microstep: 209.19 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.91 | bwd_microstep: 190.09 | bwd_inner_microstep: 189.73 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 163.14 | bwd_microstep: 189.19 | bwd_inner_microstep: 188.82 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.62 | bwd_microstep: 191.40 | bwd_inner_microstep: 191.23 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.18 | bwd_microstep: 219.81 | bwd_inner_microstep: 208.02 | bwd_allreduce_microstep: 11.63 | step_microstep: 20.82 [Rank 0] time (ms) | fwd: 1297.10 | bwd: 1662.07 | bwd_inner: 1648.35 | bwd_allreduce: 12.54 | step: 21.92 19%|█▉ | 549/2865 [3:44:30<5:39:04, 8.78s/it] {'loss': 0.7421, 'learning_rate': 3.732239189482088e-05, 'epoch': 0.57} 19%|█▉ | 549/2865 [3:44:30<5:39:04, 8.78s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.46 | bwd_microstep: 187.82 | bwd_inner_microstep: 187.46 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 161.02 | bwd_microstep: 190.85 | bwd_inner_microstep: 190.49 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 163.72 | bwd_microstep: 190.13 | bwd_inner_microstep: 190.02 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 168.22 | bwd_microstep: 189.74 | bwd_inner_microstep: 189.38 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.22 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 161.90 | bwd_microstep: 188.06 | bwd_inner_microstep: 187.85 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 910 [Rank 0] time (ms) | fwd_microstep: 163.49 | bwd_microstep: 203.90 | bwd_inner_microstep: 203.56 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.68 | bwd_microstep: 214.49 | bwd_inner_microstep: 214.31 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 169.64 | bwd_microstep: 249.91 | bwd_inner_microstep: 238.41 | bwd_allreduce_microstep: 11.34 | step_microstep: 21.02 [Rank 0] time (ms) | fwd: 1309.03 | bwd: 1615.08 | bwd_inner: 1601.89 | bwd_allreduce: 12.14 | step: 21.97 19%|█▉ | 550/2865 [3:44:33<4:32:59, 7.08s/it] {'loss': 1.0874, 'learning_rate': 3.7311079758370185e-05, 'epoch': 0.58} 19%|█▉ | 550/2865 [3:44:33<4:32:59, 7.08s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 939 [Rank 0] time (ms) | fwd_microstep: 166.07 | bwd_microstep: 214.51 | bwd_inner_microstep: 214.21 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 161.50 | bwd_microstep: 215.61 | bwd_inner_microstep: 215.25 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.93 | bwd_microstep: 215.46 | bwd_inner_microstep: 215.20 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 163.27 | bwd_microstep: 215.35 | bwd_inner_microstep: 215.22 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 167.45 | bwd_microstep: 212.74 | bwd_inner_microstep: 212.24 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 164.45 | bwd_microstep: 214.96 | bwd_inner_microstep: 214.84 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.09 | bwd_microstep: 216.52 | bwd_inner_microstep: 216.39 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.26 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 164.58 | bwd_microstep: 250.04 | bwd_inner_microstep: 236.94 | bwd_allreduce_microstep: 12.96 | step_microstep: 28.03 [Rank 0] time (ms) | fwd: 1308.24 | bwd: 1755.38 | bwd_inner: 1740.67 | bwd_allreduce: 13.73 | step: 28.80 19%|█▉ | 551/2865 [3:44:36<3:48:22, 5.92s/it] {'loss': 0.9415, 'learning_rate': 3.729974549877973e-05, 'epoch': 0.58} 19%|█▉ | 551/2865 [3:44:36<3:48:22, 5.92s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 170.98 | bwd_microstep: 213.47 | bwd_inner_microstep: 213.10 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 924 [Rank 0] time (ms) | fwd_microstep: 170.16 | bwd_microstep: 214.38 | bwd_inner_microstep: 214.24 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.57 | bwd_microstep: 215.43 | bwd_inner_microstep: 215.08 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 162.56 | bwd_microstep: 216.34 | bwd_inner_microstep: 216.16 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 160.90 | bwd_microstep: 217.12 | bwd_inner_microstep: 216.99 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 162.09 | bwd_microstep: 215.18 | bwd_inner_microstep: 214.88 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 161.93 | bwd_microstep: 215.39 | bwd_inner_microstep: 215.26 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.51 | bwd_microstep: 246.53 | bwd_inner_microstep: 236.87 | bwd_allreduce_microstep: 9.57 | step_microstep: 21.27 [Rank 0] time (ms) | fwd: 1313.55 | bwd: 1753.97 | bwd_inner: 1742.91 | bwd_allreduce: 10.26 | step: 22.02 19%|█▉ | 552/2865 [3:44:40<3:17:20, 5.12s/it] {'loss': 0.8442, 'learning_rate': 3.728838913053442e-05, 'epoch': 0.58} 19%|█▉ | 552/2865 [3:44:40<3:17:20, 5.12s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 162.45 | bwd_microstep: 199.92 | bwd_inner_microstep: 199.60 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 161.49 | bwd_microstep: 193.84 | bwd_inner_microstep: 193.47 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 161.95 | bwd_microstep: 188.45 | bwd_inner_microstep: 188.22 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.11 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.96 | bwd_microstep: 187.28 | bwd_inner_microstep: 186.96 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 160.92 | bwd_microstep: 190.99 | bwd_inner_microstep: 190.87 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 162.39 | bwd_microstep: 197.31 | bwd_inner_microstep: 196.93 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 162.78 | bwd_microstep: 191.17 | bwd_inner_microstep: 190.87 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.66 | bwd_microstep: 222.04 | bwd_inner_microstep: 207.25 | bwd_allreduce_microstep: 14.67 | step_microstep: 20.89 [Rank 0] time (ms) | fwd: 1293.48 | bwd: 1571.15 | bwd_inner: 1554.65 | bwd_allreduce: 15.50 | step: 21.71 19%|█▉ | 553/2865 [3:44:43<2:53:05, 4.49s/it] {'loss': 0.9817, 'learning_rate': 3.727701066814743e-05, 'epoch': 0.58} 19%|█▉ | 553/2865 [3:44:43<2:53:05, 4.49s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 942 [Rank 0] time (ms) | fwd_microstep: 162.42 | bwd_microstep: 193.44 | bwd_inner_microstep: 193.14 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 161.25 | bwd_microstep: 191.31 | bwd_inner_microstep: 191.20 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 165.88 | bwd_microstep: 190.53 | bwd_inner_microstep: 190.42 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 160.79 | bwd_microstep: 192.28 | bwd_inner_microstep: 192.15 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.99 | bwd_microstep: 191.40 | bwd_inner_microstep: 190.84 | bwd_allreduce_microstep: 0.23 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 164.11 | bwd_microstep: 187.29 | bwd_inner_microstep: 187.02 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.17 | bwd_microstep: 188.16 | bwd_inner_microstep: 187.94 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.43 | bwd_microstep: 214.70 | bwd_inner_microstep: 203.79 | bwd_allreduce_microstep: 10.74 | step_microstep: 20.82 [Rank 0] time (ms) | fwd: 1295.93 | bwd: 1549.26 | bwd_inner: 1536.91 | bwd_allreduce: 11.42 | step: 21.45 19%|█▉ | 554/2865 [3:44:46<2:35:49, 4.05s/it] {'loss': 1.0621, 'learning_rate': 3.7265610126160155e-05, 'epoch': 0.58} 19%|█▉ | 554/2865 [3:44:46<2:35:49, 4.05s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 164.47 | bwd_microstep: 211.47 | bwd_inner_microstep: 211.18 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 161.45 | bwd_microstep: 231.34 | bwd_inner_microstep: 231.05 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.74 | bwd_microstep: 192.48 | bwd_inner_microstep: 192.13 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.00 | bwd_microstep: 189.44 | bwd_inner_microstep: 189.15 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 161.40 | bwd_microstep: 189.52 | bwd_inner_microstep: 189.22 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.20 | bwd_microstep: 188.57 | bwd_inner_microstep: 188.22 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 164.09 | bwd_microstep: 424.09 | bwd_inner_microstep: 423.73 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.20 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.81 | bwd_microstep: 218.21 | bwd_inner_microstep: 208.08 | bwd_allreduce_microstep: 9.97 | step_microstep: 21.46 [Rank 0] time (ms) | fwd: 1296.03 | bwd: 1845.28 | bwd_inner: 1833.30 | bwd_allreduce: 10.84 | step: 22.38 19%|█▉ | 555/2865 [3:44:49<2:27:06, 3.82s/it] {'loss': 0.8558, 'learning_rate': 3.72541875191422e-05, 'epoch': 0.58} 19%|█▉ | 555/2865 [3:44:49<2:27:06, 3.82s/it][INFO|trainer.py:3242] 2026-05-15 00:05:32,091 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-15 00:05:32,091 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-15 00:05:32,091 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-555 [INFO|configuration_utils.py:473] 2026-05-15 00:10:36,999 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-555/config.json [INFO|configuration_utils.py:594] 2026-05-15 00:10:37,017 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-555/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-15 00:10:42,617 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-555/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-15 00:10:42,627 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-555/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-15 00:10:42,634 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-555/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-15 00:10:42,637 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-555/added_tokens.json [INFO|trainer.py:3028] 2026-05-15 00:10:52,363 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-480] due to args.save_total_limit dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 302010.12 | bwd_microstep: 217.27 | bwd_inner_microstep: 216.92 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.19 | bwd_microstep: 214.96 | bwd_inner_microstep: 214.83 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 159.68 | bwd_microstep: 214.68 | bwd_inner_microstep: 214.41 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 159.93 | bwd_microstep: 215.20 | bwd_inner_microstep: 214.85 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.14 | bwd_microstep: 214.82 | bwd_inner_microstep: 214.62 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 159.97 | bwd_microstep: 213.88 | bwd_inner_microstep: 213.76 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.00 | bwd_microstep: 214.04 | bwd_inner_microstep: 213.72 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.07 | bwd_microstep: 248.70 | bwd_inner_microstep: 237.80 | bwd_allreduce_microstep: 10.75 | step_microstep: 21.14 [Rank 0] time (ms) | fwd: 303122.24 | bwd: 1753.67 | bwd_inner: 1741.37 | bwd_allreduce: 11.43 | step: 21.64 19%|█▉ | 556/2865 [3:50:13<64:03:32, 99.88s/it] {'loss': 0.7757, 'learning_rate': 3.724274286169139e-05, 'epoch': 0.58} 19%|█▉ | 556/2865 [3:50:13<64:03:32, 99.88s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 160.44 | bwd_microstep: 214.28 | bwd_inner_microstep: 213.96 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 160.81 | bwd_microstep: 215.15 | bwd_inner_microstep: 215.03 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 160.26 | bwd_microstep: 213.79 | bwd_inner_microstep: 213.51 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.72 | bwd_microstep: 215.62 | bwd_inner_microstep: 215.34 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.14 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.25 | bwd_microstep: 223.56 | bwd_inner_microstep: 223.22 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.09 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 159.84 | bwd_microstep: 219.95 | bwd_inner_microstep: 219.56 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.08 | bwd_microstep: 214.78 | bwd_inner_microstep: 214.60 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.20 | bwd_microstep: 251.15 | bwd_inner_microstep: 239.78 | bwd_allreduce_microstep: 11.19 | step_microstep: 20.86 [Rank 0] time (ms) | fwd: 1282.52 | bwd: 1768.42 | bwd_inner: 1755.48 | bwd_allreduce: 11.95 | step: 21.51 19%|█▉ | 557/2865 [3:50:16<45:26:17, 70.87s/it] {'loss': 0.7697, 'learning_rate': 3.723127616843372e-05, 'epoch': 0.58} 19%|█▉ | 557/2865 [3:50:16<45:26:17, 70.87s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 935 [Rank 0] time (ms) | fwd_microstep: 161.56 | bwd_microstep: 214.05 | bwd_inner_microstep: 213.93 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 163.85 | bwd_microstep: 215.31 | bwd_inner_microstep: 214.94 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 162.97 | bwd_microstep: 214.32 | bwd_inner_microstep: 213.92 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 163.53 | bwd_microstep: 216.20 | bwd_inner_microstep: 216.08 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 165.73 | bwd_microstep: 214.66 | bwd_inner_microstep: 214.30 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 159.77 | bwd_microstep: 213.35 | bwd_inner_microstep: 213.05 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 147.24 | bwd_microstep: 187.91 | bwd_inner_microstep: 187.61 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 160.67 | bwd_microstep: 246.04 | bwd_inner_microstep: 236.17 | bwd_allreduce_microstep: 9.72 | step_microstep: 21.21 [Rank 0] time (ms) | fwd: 1285.19 | bwd: 1721.95 | bwd_inner: 1710.51 | bwd_allreduce: 10.44 | step: 21.78 19%|█▉ | 558/2865 [3:50:19<32:24:20, 50.57s/it] {'loss': 0.7373, 'learning_rate': 3.721978745402334e-05, 'epoch': 0.58} 19%|█▉ | 558/2865 [3:50:19<32:24:20, 50.57s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 937 [Rank 0] time (ms) | fwd_microstep: 161.52 | bwd_microstep: 213.77 | bwd_inner_microstep: 213.43 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 925 [Rank 0] time (ms) | fwd_microstep: 160.96 | bwd_microstep: 215.02 | bwd_inner_microstep: 214.72 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 160.67 | bwd_microstep: 222.85 | bwd_inner_microstep: 222.52 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 160.41 | bwd_microstep: 214.93 | bwd_inner_microstep: 214.63 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.61 | bwd_microstep: 214.68 | bwd_inner_microstep: 214.33 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 164.51 | bwd_microstep: 216.79 | bwd_inner_microstep: 216.45 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.47 | bwd_microstep: 215.20 | bwd_inner_microstep: 214.84 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.79 | bwd_microstep: 246.85 | bwd_inner_microstep: 236.97 | bwd_allreduce_microstep: 9.73 | step_microstep: 21.06 [Rank 0] time (ms) | fwd: 1289.82 | bwd: 1760.17 | bwd_inner: 1748.52 | bwd_allreduce: 10.59 | step: 21.58 20%|█▉ | 559/2865 [3:50:23<23:17:30, 36.36s/it] {'loss': 0.8195, 'learning_rate': 3.720827673314255e-05, 'epoch': 0.58} 20%|█▉ | 559/2865 [3:50:23<23:17:30, 36.36s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 930 [Rank 0] time (ms) | fwd_microstep: 161.46 | bwd_microstep: 213.44 | bwd_inner_microstep: 213.04 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 160.80 | bwd_microstep: 215.11 | bwd_inner_microstep: 214.72 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.33 | bwd_microstep: 212.65 | bwd_inner_microstep: 212.34 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 163.14 | bwd_microstep: 216.78 | bwd_inner_microstep: 216.33 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 160.61 | bwd_microstep: 226.53 | bwd_inner_microstep: 226.38 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.06 | bwd_microstep: 217.09 | bwd_inner_microstep: 216.73 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.17 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 909 [Rank 0] time (ms) | fwd_microstep: 160.03 | bwd_microstep: 216.37 | bwd_inner_microstep: 216.01 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 161.06 | bwd_microstep: 251.09 | bwd_inner_microstep: 241.24 | bwd_allreduce_microstep: 9.63 | step_microstep: 21.00 [Rank 0] time (ms) | fwd: 1287.39 | bwd: 1769.22 | bwd_inner: 1757.41 | bwd_allreduce: 10.65 | step: 21.87 20%|█▉ | 560/2865 [3:50:26<16:54:52, 26.42s/it] {'loss': 0.8909, 'learning_rate': 3.719674402050177e-05, 'epoch': 0.59} 20%|█▉ | 560/2865 [3:50:26<16:54:52, 26.42s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 161.09 | bwd_microstep: 214.20 | bwd_inner_microstep: 214.07 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 161.24 | bwd_microstep: 215.65 | bwd_inner_microstep: 215.40 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 165.03 | bwd_microstep: 224.05 | bwd_inner_microstep: 223.74 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.86 | bwd_microstep: 216.98 | bwd_inner_microstep: 216.47 | bwd_allreduce_microstep: 0.20 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 164.15 | bwd_microstep: 214.89 | bwd_inner_microstep: 214.50 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.49 | bwd_microstep: 215.60 | bwd_inner_microstep: 215.24 | bwd_allreduce_microstep: 0.12 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 162.95 | bwd_microstep: 212.50 | bwd_inner_microstep: 212.38 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.82 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.26 | bwd_microstep: 248.51 | bwd_inner_microstep: 237.46 | bwd_allreduce_microstep: 10.90 | step_microstep: 20.79 [Rank 0] time (ms) | fwd: 1295.95 | bwd: 1762.51 | bwd_inner: 1749.76 | bwd_allreduce: 11.67 | step: 21.43 20%|█▉ | 561/2865 [3:50:29<12:27:17, 19.46s/it] {'loss': 0.7498, 'learning_rate': 3.718518933083954e-05, 'epoch': 0.59} 20%|█▉ | 561/2865 [3:50:29<12:27:17, 19.46s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.87 | bwd_microstep: 215.43 | bwd_inner_microstep: 215.07 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.21 | bwd_microstep: 216.58 | bwd_inner_microstep: 216.28 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.14 | bwd_microstep: 216.71 | bwd_inner_microstep: 216.42 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.63 | bwd_microstep: 215.34 | bwd_inner_microstep: 215.08 | bwd_allreduce_microstep: 0.10 | step_microstep: 0.13 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.19 | bwd_microstep: 217.18 | bwd_inner_microstep: 216.88 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 159.84 | bwd_microstep: 217.60 | bwd_inner_microstep: 217.28 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 160.60 | bwd_microstep: 215.35 | bwd_inner_microstep: 215.20 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 160.18 | bwd_microstep: 246.23 | bwd_inner_microstep: 236.68 | bwd_allreduce_microstep: 9.47 | step_microstep: 20.82 [Rank 0] time (ms) | fwd: 1282.57 | bwd: 1760.54 | bwd_inner: 1749.36 | bwd_allreduce: 10.29 | step: 21.52 20%|█▉ | 562/2865 [3:50:32<9:19:45, 14.58s/it] {'loss': 0.6805, 'learning_rate': 3.717361267892245e-05, 'epoch': 0.59} 20%|█▉ | 562/2865 [3:50:32<9:19:45, 14.58s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 942 [Rank 0] time (ms) | fwd_microstep: 161.69 | bwd_microstep: 217.34 | bwd_inner_microstep: 217.00 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 160.57 | bwd_microstep: 216.43 | bwd_inner_microstep: 216.08 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.19 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.52 | bwd_microstep: 215.50 | bwd_inner_microstep: 215.33 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 162.33 | bwd_microstep: 216.60 | bwd_inner_microstep: 216.28 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.18 | bwd_microstep: 215.99 | bwd_inner_microstep: 215.69 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 160.17 | bwd_microstep: 216.03 | bwd_inner_microstep: 215.77 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.37 | bwd_microstep: 213.49 | bwd_inner_microstep: 213.16 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 146.64 | bwd_microstep: 200.09 | bwd_inner_microstep: 190.34 | bwd_allreduce_microstep: 9.58 | step_microstep: 21.06 [Rank 0] time (ms) | fwd: 1272.36 | bwd: 1711.57 | bwd_inner: 1700.16 | bwd_allreduce: 10.42 | step: 21.70 20%|█▉ | 563/2865 [3:50:35<7:07:52, 11.15s/it] {'loss': 0.7792, 'learning_rate': 3.71620140795452e-05, 'epoch': 0.59} 20%|█▉ | 563/2865 [3:50:35<7:07:52, 11.15s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 165.37 | bwd_microstep: 189.58 | bwd_inner_microstep: 189.15 | bwd_allreduce_microstep: 0.18 | step_microstep: 0.27 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 162.64 | bwd_microstep: 188.91 | bwd_inner_microstep: 188.81 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 160.98 | bwd_microstep: 190.71 | bwd_inner_microstep: 190.37 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.15 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 161.06 | bwd_microstep: 189.75 | bwd_inner_microstep: 189.46 | bwd_allreduce_microstep: 0.14 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 927 [Rank 0] time (ms) | fwd_microstep: 160.88 | bwd_microstep: 190.38 | bwd_inner_microstep: 190.08 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 921 [Rank 0] time (ms) | fwd_microstep: 183.79 | bwd_microstep: 189.24 | bwd_inner_microstep: 189.07 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 173.38 | bwd_microstep: 190.02 | bwd_inner_microstep: 189.83 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.10 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.65 | bwd_microstep: 216.19 | bwd_inner_microstep: 205.68 | bwd_allreduce_microstep: 10.38 | step_microstep: 21.17 [Rank 0] time (ms) | fwd: 1328.65 | bwd: 1544.94 | bwd_inner: 1532.91 | bwd_allreduce: 11.07 | step: 21.98 20%|█▉ | 564/2865 [3:50:38<5:35:07, 8.74s/it] {'loss': 0.7878, 'learning_rate': 3.715039354753051e-05, 'epoch': 0.59} 20%|█▉ | 564/2865 [3:50:38<5:35:07, 8.74s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 161.40 | bwd_microstep: 189.39 | bwd_inner_microstep: 189.08 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 162.13 | bwd_microstep: 191.65 | bwd_inner_microstep: 191.42 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 160.92 | bwd_microstep: 189.97 | bwd_inner_microstep: 189.53 | bwd_allreduce_microstep: 0.19 | step_microstep: 0.23 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 163.38 | bwd_microstep: 188.98 | bwd_inner_microstep: 188.77 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.70 | bwd_microstep: 187.13 | bwd_inner_microstep: 186.93 | bwd_allreduce_microstep: 0.09 | step_microstep: 0.12 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.83 | bwd_microstep: 188.88 | bwd_inner_microstep: 188.75 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 159.75 | bwd_microstep: 189.32 | bwd_inner_microstep: 189.20 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.86 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.48 | bwd_microstep: 215.91 | bwd_inner_microstep: 206.42 | bwd_allreduce_microstep: 9.41 | step_microstep: 20.82 [Rank 0] time (ms) | fwd: 1290.49 | bwd: 1541.35 | bwd_inner: 1530.50 | bwd_allreduce: 10.02 | step: 21.53 20%|█▉ | 565/2865 [3:50:41<4:28:55, 7.02s/it] {'loss': 0.9038, 'learning_rate': 3.7138751097729134e-05, 'epoch': 0.59} 20%|█▉ | 565/2865 [3:50:41<4:28:55, 7.02s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 161.82 | bwd_microstep: 189.48 | bwd_inner_microstep: 189.35 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 933 [Rank 0] time (ms) | fwd_microstep: 161.14 | bwd_microstep: 190.31 | bwd_inner_microstep: 189.94 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 917 [Rank 0] time (ms) | fwd_microstep: 160.54 | bwd_microstep: 190.28 | bwd_inner_microstep: 189.96 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 160.79 | bwd_microstep: 189.40 | bwd_inner_microstep: 189.11 | bwd_allreduce_microstep: 0.13 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 920 [Rank 0] time (ms) | fwd_microstep: 159.95 | bwd_microstep: 188.71 | bwd_inner_microstep: 188.59 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 146.82 | bwd_microstep: 173.69 | bwd_inner_microstep: 173.60 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.15 | bwd_microstep: 188.74 | bwd_inner_microstep: 188.67 | bwd_allreduce_microstep: 0.03 | step_microstep: 0.06 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 160.99 | bwd_microstep: 212.54 | bwd_inner_microstep: 202.81 | bwd_allreduce_microstep: 9.58 | step_microstep: 21.17 [Rank 0] time (ms) | fwd: 1272.07 | bwd: 1523.26 | bwd_inner: 1512.36 | bwd_allreduce: 10.11 | step: 21.78 20%|█▉ | 566/2865 [3:50:44<3:42:05, 5.80s/it] {'loss': 0.7687, 'learning_rate': 3.712708674501985e-05, 'epoch': 0.59} 20%|█▉ | 566/2865 [3:50:44<3:42:05, 5.80s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 936 [Rank 0] time (ms) | fwd_microstep: 162.03 | bwd_microstep: 190.80 | bwd_inner_microstep: 190.48 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 942 [Rank 0] time (ms) | fwd_microstep: 162.35 | bwd_microstep: 191.41 | bwd_inner_microstep: 191.07 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 926 [Rank 0] time (ms) | fwd_microstep: 161.20 | bwd_microstep: 190.06 | bwd_inner_microstep: 189.95 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.03 | bwd_microstep: 189.05 | bwd_inner_microstep: 188.95 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.98 | bwd_microstep: 189.06 | bwd_inner_microstep: 188.76 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 160.62 | bwd_microstep: 188.16 | bwd_inner_microstep: 188.01 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 160.42 | bwd_microstep: 189.33 | bwd_inner_microstep: 188.95 | bwd_allreduce_microstep: 0.17 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.89 [Rank 0] time (ms) | fwd_microstep: 160.46 | bwd_microstep: 219.65 | bwd_inner_microstep: 210.08 | bwd_allreduce_microstep: 9.48 | step_microstep: 20.81 [Rank 0] time (ms) | fwd: 1288.98 | bwd: 1547.61 | bwd_inner: 1536.58 | bwd_allreduce: 10.25 | step: 21.41 20%|█▉ | 567/2865 [3:50:47<3:09:46, 4.96s/it] {'loss': 0.8616, 'learning_rate': 3.711540050430941e-05, 'epoch': 0.59} 20%|█▉ | 567/2865 [3:50:47<3:09:46, 4.96s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 934 [Rank 0] time (ms) | fwd_microstep: 161.86 | bwd_microstep: 187.37 | bwd_inner_microstep: 187.15 | bwd_allreduce_microstep: 0.11 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | fwd_microstep: 161.28 | bwd_microstep: 189.06 | bwd_inner_microstep: 188.94 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 922 [Rank 0] time (ms) | fwd_microstep: 161.38 | bwd_microstep: 188.79 | bwd_inner_microstep: 188.41 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.16 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 923 [Rank 0] time (ms) | fwd_microstep: 161.64 | bwd_microstep: 198.12 | bwd_inner_microstep: 197.97 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 160.95 | bwd_microstep: 188.09 | bwd_inner_microstep: 187.85 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 163.36 | bwd_microstep: 190.52 | bwd_inner_microstep: 190.15 | bwd_allreduce_microstep: 0.16 | step_microstep: 0.21 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 160.74 | bwd_microstep: 188.40 | bwd_inner_microstep: 188.28 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 918 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.87 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 163.14 | bwd_microstep: 210.51 | bwd_inner_microstep: 200.87 | bwd_allreduce_microstep: 9.48 | step_microstep: 20.86 [Rank 0] time (ms) | fwd: 1294.25 | bwd: 1540.99 | bwd_inner: 1530.02 | bwd_allreduce: 10.11 | step: 21.57 20%|█▉ | 568/2865 [3:50:50<2:47:10, 4.37s/it] {'loss': 1.0138, 'learning_rate': 3.710369239053255e-05, 'epoch': 0.59} 20%|█▉ | 568/2865 [3:50:50<2:47:10, 4.37s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 161.21 | bwd_microstep: 188.48 | bwd_inner_microstep: 188.36 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 161.06 | bwd_microstep: 189.85 | bwd_inner_microstep: 189.72 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | fwd_microstep: 160.22 | bwd_microstep: 215.80 | bwd_inner_microstep: 215.51 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.08 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 913 [Rank 0] time (ms) | fwd_microstep: 160.81 | bwd_microstep: 242.68 | bwd_inner_microstep: 242.32 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 916 [Rank 0] time (ms) | fwd_microstep: 161.16 | bwd_microstep: 244.82 | bwd_inner_microstep: 244.48 | bwd_allreduce_microstep: 0.08 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 914 [Rank 0] time (ms) | fwd_microstep: 181.26 | bwd_microstep: 226.14 | bwd_inner_microstep: 225.55 | bwd_allreduce_microstep: 0.26 | step_microstep: 0.25 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | fwd_microstep: 179.20 | bwd_microstep: 222.94 | bwd_inner_microstep: 222.82 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 912 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.87 [Rank 0] time (ms) | fwd_microstep: 160.08 | bwd_microstep: 257.39 | bwd_inner_microstep: 247.65 | bwd_allreduce_microstep: 9.60 | step_microstep: 20.80 [Rank 0] time (ms) | fwd: 1324.89 | bwd: 1788.24 | bwd_inner: 1776.88 | bwd_allreduce: 10.36 | step: 21.62 20%|█▉ | 569/2865 [3:50:54<2:34:36, 4.04s/it] {'loss': 0.9101, 'learning_rate': 3.709196241865195e-05, 'epoch': 0.6} 20%|█▉ | 569/2865 [3:50:54<2:34:36, 4.04s/it]dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 955 [Rank 0] time (ms) | fwd_microstep: 162.65 | bwd_microstep: 225.71 | bwd_inner_microstep: 225.58 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 932 [Rank 0] time (ms) | fwd_microstep: 161.75 | bwd_microstep: 222.60 | bwd_inner_microstep: 222.47 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 931 [Rank 0] time (ms) | fwd_microstep: 161.76 | bwd_microstep: 220.86 | bwd_inner_microstep: 220.73 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 928 [Rank 0] time (ms) | fwd_microstep: 160.67 | bwd_microstep: 230.90 | bwd_inner_microstep: 230.77 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 929 [Rank 0] time (ms) | fwd_microstep: 161.89 | bwd_microstep: 216.55 | bwd_inner_microstep: 216.43 | bwd_allreduce_microstep: 0.05 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 919 [Rank 0] time (ms) | fwd_microstep: 160.69 | bwd_microstep: 218.30 | bwd_inner_microstep: 217.94 | bwd_allreduce_microstep: 0.15 | step_microstep: 0.18 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 915 [Rank 0] time (ms) | fwd_microstep: 162.20 | bwd_microstep: 218.17 | bwd_inner_microstep: 218.02 | bwd_allreduce_microstep: 0.06 | step_microstep: 0.07 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 911 [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.83 | optimizer_step: 4.88 [Rank 0] time (ms) | fwd_microstep: 162.54 | bwd_microstep: 247.81 | bwd_inner_microstep: 238.33 | bwd_allreduce_microstep: 9.41 | step_microstep: 21.12 [Rank 0] time (ms) | fwd: 1294.02 | bwd: 1801.01 | bwd_inner: 1790.52 | bwd_allreduce: 9.90 | step: 21.74 20%|█▉ | 570/2865 [3:50:57<2:25:31, 3.80s/it] {'loss': 0.846, 'learning_rate': 3.708021060365824e-05, 'epoch': 0.6} 20%|█▉ | 570/2865 [3:50:57<2:25:31, 3.80s/it][INFO|trainer.py:3242] 2026-05-15 00:11:40,063 >> ***** Running Evaluation ***** [INFO|trainer.py:3244] 2026-05-15 00:11:40,063 >> Num examples = 1700 [INFO|trainer.py:3247] 2026-05-15 00:11:40,063 >> Batch size = 2 dynamic ViT batch size: 6, images per sample: 3.0, dynamic token length: 4096 0%| | 0/850 [00:00> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-570 [INFO|configuration_utils.py:473] 2026-05-15 00:16:44,329 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-570/config.json [INFO|configuration_utils.py:594] 2026-05-15 00:16:44,341 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-570/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-15 00:16:50,159 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-570/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-15 00:16:50,170 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-570/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-15 00:16:50,178 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-570/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-15 00:16:50,181 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/tmp-checkpoint-570/added_tokens.json [INFO|trainer.py:3028] 2026-05-15 00:17:00,889 >> Deleting older checkpoint [work_dirs/internvl3_1b_walk_lora/checkpoint-510] due to args.save_total_limit [INFO|trainer.py:1962] 2026-05-15 00:17:01,582 >> Training completed. Do not forget to share your model on huggingface.co/models =) [INFO|trainer.py:2160] 2026-05-15 00:17:01,583 >> Loading best model from work_dirs/internvl3_1b_walk_lora/checkpoint-495 (score: 0.9165481328964233). [INFO|deepspeed.py:400] 2026-05-15 00:17:01,586 >> Attempting to resume from work_dirs/internvl3_1b_walk_lora/checkpoint-495 {'train_runtime': 14191.3031, 'train_samples_per_second': 3.233, 'train_steps_per_second': 0.202, 'train_loss': 1.079708440157405, 'epoch': 0.6} 20%|█▉ | 570/2865 [3:56:29<2:25:31, 3.80s/it] 20%|█▉ | 570/2865 [3:56:29<15:52:09, 24.89s/it] [INFO|trainer.py:2936] 2026-05-15 00:17:12,339 >> Saving model checkpoint to work_dirs/internvl3_1b_walk_lora [INFO|configuration_utils.py:473] 2026-05-15 00:17:12,350 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/config.json [INFO|configuration_utils.py:594] 2026-05-15 00:17:12,360 >> Configuration saved in work_dirs/internvl3_1b_walk_lora/generation_config.json [INFO|modeling_utils.py:2493] 2026-05-15 00:17:17,984 >> Model weights saved in work_dirs/internvl3_1b_walk_lora/model.safetensors [INFO|tokenization_utils_base.py:2433] 2026-05-15 00:17:17,995 >> tokenizer config file saved in work_dirs/internvl3_1b_walk_lora/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2026-05-15 00:17:18,009 >> Special tokens file saved in work_dirs/internvl3_1b_walk_lora/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2026-05-15 00:17:18,012 >> added tokens file saved in work_dirs/internvl3_1b_walk_lora/added_tokens.json ***** train metrics ***** epoch = 0.6 train_loss = 1.0797 train_runtime = 3:56:31.30 train_samples = 15294 train_samples_per_second = 3.233 train_steps_per_second = 0.202 wandb: wandb: 🚀 View run InternVL3-1B_walkvlm_lora_earlystop at: https://wandb.ai/vlm-blind-assist/internvl-walk/runs/ampnlkg8 wandb: Find logs at: wandb/run-20260514_202041-ampnlkg8/logs