W0224 00:42:59.265000 67984 site-packages/torch/distributed/run.py:792] 
W0224 00:42:59.265000 67984 site-packages/torch/distributed/run.py:792] *****************************************
W0224 00:42:59.265000 67984 site-packages/torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0224 00:42:59.265000 67984 site-packages/torch/distributed/run.py:792] *****************************************
[2026-02-24 00:43:02,299] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:43:02,300] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:43:02,304] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:43:02,304] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:43:02,311] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:43:02,317] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:43:02,317] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:43:02,318] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.petrel_client is not installed. If you read data locally instead of from ceph, ignore it.

petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. Using PIL to load images.
[2026-02-24 00:43:11,029] [INFO] [comm.py:652:init_distributed] cdb=None
[2026-02-24 00:43:11,029] [INFO] [comm.py:683:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
02/24/2026 00:43:11 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False
02/24/2026 00:43:11 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
_n_gpu=1,
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=False,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_steps=None,
evaluation_strategy=IntervalStrategy.NO,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=2,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=False,
hub_strategy=HubStrategy.EVERY_SAVE,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=2e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/mnt/shared-storage-user/jiaziheng/LMMS/internvl-pretrain-02_23_full-yt-gaming-8B/runs/Feb24_00-43-11_gpu-lg-cmc-h-h200-0803.host.h.pjlab.org.cn,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1.0,
logging_strategy=IntervalStrategy.STEPS,
lr_scheduler_kwargs={},
lr_scheduler_type=SchedulerType.COSINE,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=3.0,
optim=OptimizerNames.ADAMW_TORCH,
optim_args=None,
output_dir=/mnt/shared-storage-user/jiaziheng/LMMS/internvl-pretrain-02_23_full-yt-gaming-8B,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=8,
per_device_train_batch_size=1,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
resume_from_checkpoint=None,
run_name=/mnt/shared-storage-user/jiaziheng/LMMS/internvl-pretrain-02_23_full-yt-gaming-8B,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=1000,
save_strategy=IntervalStrategy.STEPS,
save_total_limit=1,
seed=42,
skip_memory_metrics=True,
split_batches=False,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.05,
)
[2026-02-24 00:43:11,680] [INFO] [comm.py:652:init_distributed] cdb=None
[2026-02-24 00:43:11,681] [INFO] [comm.py:652:init_distributed] cdb=None
02/24/2026 00:43:11 - INFO - __main__ - Loading Tokenizer: /mnt/shared-storage-user/jiaziheng/LMMS/internvl-pretrain-10_9_clip
[2026-02-24 00:43:11,684] [INFO] [comm.py:652:init_distributed] cdb=None
[2026-02-24 00:43:11,685] [INFO] [comm.py:652:init_distributed] cdb=None
[2026-02-24 00:43:11,689] [INFO] [comm.py:652:init_distributed] cdb=None
[2026-02-24 00:43:11,692] [INFO] [comm.py:652:init_distributed] cdb=None
[2026-02-24 00:43:11,697] [INFO] [comm.py:652:init_distributed] cdb=None
[INFO|tokenization_utils_base.py:2025] 2026-02-24 00:43:11,709 >> loading file vocab.json
[INFO|tokenization_utils_base.py:2025] 2026-02-24 00:43:11,709 >> loading file merges.txt
[INFO|tokenization_utils_base.py:2025] 2026-02-24 00:43:11,709 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2025] 2026-02-24 00:43:11,709 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2025] 2026-02-24 00:43:11,709 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2025] 2026-02-24 00:43:11,709 >> loading file tokenizer.json
[WARNING|logging.py:314] 2026-02-24 00:43:11,906 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
02/24/2026 00:43:11 - INFO - __main__ - Loading InternVLChatModel...
[INFO|configuration_utils.py:727] 2026-02-24 00:43:11,911 >> loading configuration file /mnt/shared-storage-user/jiaziheng/LMMS/internvl-pretrain-10_9_clip/config.json
[INFO|configuration_utils.py:792] 2026-02-24 00:43:11,912 >> Model config InternVLChatConfig {
  "_commit_hash": null,
  "_name_or_path": "/mnt/shared-storage-user/jiaziheng/LMMS/InternVL3-8B-Instruct",
  "architectures": [
    "InternVLChatModel"
  ],
  "auto_map": {
    "AutoConfig": "configuration_internvl_chat.InternVLChatConfig",
    "AutoModel": "modeling_internvl_chat.InternVLChatModel",
    "AutoModelForCausalLM": "modeling_internvl_chat.InternVLChatModel"
  },
  "downsample_ratio": 0.5,
  "dynamic_image_size": true,
  "force_image_size": 448,
  "hidden_size": 3584,
  "image_fold": null,
  "llm_config": {
    "_name_or_path": "./pretrained/Qwen2.5-32B-Instruct",
    "add_cross_attention": false,
    "architectures": [
      "Qwen2ForCausalLM"
    ],
    "attention_dropout": 0.0,
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": 151643,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": 151643,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "silu",
    "hidden_size": 3584,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "initializer_range": 0.02,
    "intermediate_size": 18944,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "length_penalty": 1.0,
    "max_length": 20,
    "max_position_embeddings": 32768,
    "max_window_layers": 70,
    "min_length": 0,
    "model_type": "qwen2",
    "moe_config": null,
    "no_repeat_ngram_size": 0,
    "num_attention_heads": 28,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_hidden_layers": 28,
    "num_key_value_heads": 4,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": true,
    "output_scores": false,
    "pad_token_id": null,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "rms_norm_eps": 1e-06,
    "rope_scaling": {
      "factor": 2.0,
      "rope_type": "dynamic",
      "type": "dynamic"
    },
    "rope_theta": 1000000.0,
    "sep_token_id": null,
    "sliding_window": null,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": false,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.37.2",
    "typical_p": 1.0,
    "use_bfloat16": true,
    "use_cache": false,
    "use_sliding_window": false,
    "vocab_size": 151674
  },
  "max_dynamic_patch": 12,
  "min_dynamic_patch": 1,
  "model_type": "internvl_chat",
  "num_detectors": 5,
  "num_obj_queries": 3,
  "pad2square": false,
  "ps_version": "v2",
  "select_layer": -1,
  "template": "internvl2_5",
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": null,
  "use_backbone_lora": 0,
  "use_llm_lora": 0,
  "use_thumbnail": true,
  "vision_config": {
    "_name_or_path": "OpenGVLab/InternViT-6B-448px-V1-5",
    "add_cross_attention": false,
    "architectures": [
      "InternVisionModel"
    ],
    "attention_dropout": 0.0,
    "auto_map": {
      "AutoConfig": "configuration_intern_vit.InternVisionConfig",
      "AutoModel": "modeling_intern_vit.InternVisionModel"
    },
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": null,
    "capacity_factor": 1.2,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "drop_path_rate": 0.0,
    "dropout": 0.0,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": null,
    "eval_capacity_factor": 1.4,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "gelu",
    "hidden_size": 1024,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "image_size": 448,
    "initializer_factor": 0.1,
    "initializer_range": 1e-10,
    "intermediate_size": 4096,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "laux_allreduce": "all_nodes",
    "layer_norm_eps": 1e-06,
    "length_penalty": 1.0,
    "max_length": 20,
    "min_length": 0,
    "model_type": "intern_vit_6b",
    "moe_coeff_ratio": 0.5,
    "moe_intermediate_size": 768,
    "moe_output_scale": 4.0,
    "no_repeat_ngram_size": 0,
    "noisy_gate_policy": "RSample_before",
    "norm_type": "layer_norm",
    "num_attention_heads": 16,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_channels": 3,
    "num_experts": 8,
    "num_hidden_layers": 24,
    "num_return_sequences": 1,
    "num_routed_experts": 4,
    "num_shared_experts": 4,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "patch_size": 14,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "qk_normalization": false,
    "qkv_bias": true,
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "sep_token_id": null,
    "shared_expert_intermediate_size": 3072,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": true,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.37.2",
    "typical_p": 1.0,
    "use_bfloat16": true,
    "use_flash_attn": true,
    "use_moe": false,
    "use_residual": true,
    "use_rts": false,
    "use_weighted_residual": false
  }
}

02/24/2026 00:43:11 - INFO - __main__ - Using flash_attention_2 for LLaMA
[INFO|modeling_utils.py:3473] 2026-02-24 00:43:11,916 >> loading weights file /mnt/shared-storage-user/jiaziheng/LMMS/internvl-pretrain-10_9_clip/model.safetensors.index.json
[INFO|modeling_utils.py:1426] 2026-02-24 00:43:11,917 >> Instantiating InternVLChatModel model under default dtype torch.bfloat16.
[INFO|configuration_utils.py:826] 2026-02-24 00:43:11,918 >> Generate config GenerationConfig {}

[INFO|configuration_utils.py:826] 2026-02-24 00:43:11,954 >> Generate config GenerationConfig {
  "bos_token_id": 151643,
  "eos_token_id": 151643,
  "output_hidden_states": true,
  "use_cache": false
}

02/24/2026 00:43:11 - WARNING - __main__ - Process rank: 5, device: cuda:5, n_gpu: 1distributed training: True, 16-bits training: False
02/24/2026 00:43:11 - WARNING - __main__ - Process rank: 3, device: cuda:3, n_gpu: 1distributed training: True, 16-bits training: False
02/24/2026 00:43:11 - WARNING - __main__ - Process rank: 2, device: cuda:2, n_gpu: 1distributed training: True, 16-bits training: False
02/24/2026 00:43:11 - WARNING - __main__ - Process rank: 1, device: cuda:1, n_gpu: 1distributed training: True, 16-bits training: False
02/24/2026 00:43:11 - WARNING - __main__ - Process rank: 4, device: cuda:4, n_gpu: 1distributed training: True, 16-bits training: False
02/24/2026 00:43:11 - WARNING - __main__ - Process rank: 7, device: cuda:7, n_gpu: 1distributed training: True, 16-bits training: False
02/24/2026 00:43:11 - WARNING - __main__ - Process rank: 6, device: cuda:6, n_gpu: 1distributed training: True, 16-bits training: False
[WARNING|logging.py:314] 2026-02-24 00:43:12,134 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[WARNING|logging.py:314] 2026-02-24 00:43:12,145 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[WARNING|logging.py:314] 2026-02-24 00:43:12,146 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[WARNING|logging.py:314] 2026-02-24 00:43:12,150 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[WARNING|logging.py:314] 2026-02-24 00:43:12,157 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[WARNING|logging.py:314] 2026-02-24 00:43:12,244 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[WARNING|logging.py:314] 2026-02-24 00:43:12,247 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:20<01:01, 20.49s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:20<01:02, 20.71s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:20<01:02, 20.72s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:20<01:02, 20.71s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:20<01:01, 20.60s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:20<01:02, 20.88s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:20<01:02, 20.74s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:20<01:02, 20.76s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:39<00:39, 19.70s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:39<00:39, 19.63s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:39<00:39, 19.70s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:39<00:39, 19.72s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:39<00:39, 19.78s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:39<00:39, 19.76s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:39<00:39, 19.73s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:39<00:39, 19.69s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:59<00:19, 19.58s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:59<00:19, 19.58s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:59<00:19, 19.64s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:59<00:19, 19.62s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:59<00:19, 19.63s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:59<00:19, 19.58s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:59<00:19, 19.59s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:59<00:19, 19.63s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [01:03<00:00, 13.73s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [01:03<00:00, 15.97s/it]
Total parameters: ~7991.79 MB)
Trainable parameters: ~7991.79 MB)
data_args.use_packed_ds False

Loading checkpoint shards: 100%|██████████| 4/4 [01:03<00:00, 13.73s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [01:03<00:00, 15.99s/it]

Loading checkpoint shards: 100%|██████████| 4/4 [01:04<00:00, 13.75s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [01:04<00:00, 16.02s/it]
[INFO|modeling_utils.py:4350] 2026-02-24 00:44:17,313 >> All model checkpoint weights were used when initializing InternVLChatModel.

[INFO|modeling_utils.py:4358] 2026-02-24 00:44:17,313 >> All the weights of InternVLChatModel were initialized from the model checkpoint at /mnt/shared-storage-user/jiaziheng/LMMS/internvl-pretrain-10_9_clip.
If your task is similar to the task the model of the checkpoint was trained on, you can already use InternVLChatModel for predictions without further training.

Loading checkpoint shards: 100%|██████████| 4/4 [01:03<00:00, 13.73s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [01:03<00:00, 15.98s/it]
[INFO|configuration_utils.py:779] 2026-02-24 00:44:17,318 >> loading configuration file /mnt/shared-storage-user/jiaziheng/LMMS/internvl-pretrain-10_9_clip/generation_config.json
Total parameters: ~7991.79 MB)
Trainable parameters: ~7991.79 MB)
[INFO|configuration_utils.py:826] 2026-02-24 00:44:17,318 >> Generate config GenerationConfig {}

data_args.use_packed_ds False
02/24/2026 00:44:17 - INFO - __main__ - Finished
02/24/2026 00:44:17 - INFO - __main__ - model.config.force_image_size: 448
02/24/2026 00:44:17 - INFO - __main__ - data_args.force_image_size: 448
02/24/2026 00:44:17 - INFO - __main__ - model.config.vision_config.image_size: 448
02/24/2026 00:44:17 - INFO - __main__ - [Dataset] num_image_token: 256
02/24/2026 00:44:17 - INFO - __main__ - [Dataset] dynamic_image_size: True
02/24/2026 00:44:17 - INFO - __main__ - [Dataset] use_thumbnail: True
02/24/2026 00:44:17 - INFO - __main__ - [Dataset] min_dynamic_patch: 1, max_dynamic_patch: 12
02/24/2026 00:44:17 - INFO - __main__ - Formatting inputs...Skip in lazy mode
02/24/2026 00:44:17 - INFO - __main__ - Add dataset: yt-gaming_intra with length: 478
02/24/2026 00:44:17 - INFO - __main__ - vision_model.embeddings.class_embedding
02/24/2026 00:44:17 - INFO - __main__ - vision_model.embeddings.position_embedding
02/24/2026 00:44:17 - INFO - __main__ - vision_model.embeddings.patch_embedding.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.embeddings.patch_embedding.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.0.ls1
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.0.ls2
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.0.attn.qkv.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.0.attn.qkv.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.0.attn.proj.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.0.attn.proj.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.0.mlp.fc1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.0.mlp.fc1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.0.mlp.fc2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.0.mlp.fc2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.0.norm1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.0.norm1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.0.norm2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.0.norm2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.1.ls1
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.1.ls2
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.1.attn.qkv.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.1.attn.qkv.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.1.attn.proj.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.1.attn.proj.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.1.mlp.fc1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.1.mlp.fc1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.1.mlp.fc2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.1.mlp.fc2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.1.norm1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.1.norm1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.1.norm2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.1.norm2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.2.ls1
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.2.ls2
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.2.attn.qkv.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.2.attn.qkv.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.2.attn.proj.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.2.attn.proj.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.2.mlp.fc1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.2.mlp.fc1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.2.mlp.fc2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.2.mlp.fc2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.2.norm1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.2.norm1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.2.norm2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.2.norm2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.3.ls1
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.3.ls2
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.3.attn.qkv.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.3.attn.qkv.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.3.attn.proj.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.3.attn.proj.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.3.mlp.fc1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.3.mlp.fc1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.3.mlp.fc2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.3.mlp.fc2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.3.norm1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.3.norm1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.3.norm2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.3.norm2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.4.ls1
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.4.ls2
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.4.attn.qkv.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.4.attn.qkv.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.4.attn.proj.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.4.attn.proj.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.4.mlp.fc1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.4.mlp.fc1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.4.mlp.fc2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.4.mlp.fc2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.4.norm1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.4.norm1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.4.norm2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.4.norm2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.5.ls1
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.5.ls2
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.5.attn.qkv.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.5.attn.qkv.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.5.attn.proj.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.5.attn.proj.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.5.mlp.fc1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.5.mlp.fc1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.5.mlp.fc2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.5.mlp.fc2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.5.norm1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.5.norm1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.5.norm2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.5.norm2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.6.ls1
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.6.ls2
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.6.attn.qkv.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.6.attn.qkv.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.6.attn.proj.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.6.attn.proj.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.6.mlp.fc1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.6.mlp.fc1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.6.mlp.fc2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.6.mlp.fc2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.6.norm1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.6.norm1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.6.norm2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.6.norm2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.7.ls1
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.7.ls2
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.7.attn.qkv.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.7.attn.qkv.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.7.attn.proj.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.7.attn.proj.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.7.mlp.fc1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.7.mlp.fc1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.7.mlp.fc2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.7.mlp.fc2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.7.norm1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.7.norm1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.7.norm2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.7.norm2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.8.ls1
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.8.ls2
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.8.attn.qkv.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.8.attn.qkv.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.8.attn.proj.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.8.attn.proj.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.8.mlp.fc1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.8.mlp.fc1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.8.mlp.fc2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.8.mlp.fc2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.8.norm1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.8.norm1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.8.norm2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.8.norm2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.9.ls1
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.9.ls2
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.9.attn.qkv.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.9.attn.qkv.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.9.attn.proj.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.9.attn.proj.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.9.mlp.fc1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.9.mlp.fc1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.9.mlp.fc2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.9.mlp.fc2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.9.norm1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.9.norm1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.9.norm2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.9.norm2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.10.ls1
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.10.ls2
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.10.attn.qkv.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.10.attn.qkv.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.10.attn.proj.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.10.attn.proj.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.10.mlp.fc1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.10.mlp.fc1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.10.mlp.fc2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.10.mlp.fc2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.10.norm1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.10.norm1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.10.norm2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.10.norm2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.11.ls1
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.11.ls2
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.11.attn.qkv.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.11.attn.qkv.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.11.attn.proj.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.11.attn.proj.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.11.mlp.fc1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.11.mlp.fc1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.11.mlp.fc2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.11.mlp.fc2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.11.norm1.weight

Loading checkpoint shards: 100%|██████████| 4/4 [01:03<00:00, 13.74s/it]02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.11.norm1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.11.norm2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.11.norm2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.12.ls1
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.12.ls2
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.12.attn.qkv.weight

Loading checkpoint shards: 100%|██████████| 4/4 [01:03<00:00, 15.99s/it]
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.12.attn.qkv.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.12.attn.proj.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.12.attn.proj.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.12.mlp.fc1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.12.mlp.fc1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.12.mlp.fc2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.12.mlp.fc2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.12.norm1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.12.norm1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.12.norm2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.12.norm2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.13.ls1
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.13.ls2
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.13.attn.qkv.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.13.attn.qkv.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.13.attn.proj.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.13.attn.proj.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.13.mlp.fc1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.13.mlp.fc1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.13.mlp.fc2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.13.mlp.fc2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.13.norm1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.13.norm1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.13.norm2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.13.norm2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.14.ls1
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.14.ls2
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.14.attn.qkv.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.14.attn.qkv.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.14.attn.proj.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.14.attn.proj.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.14.mlp.fc1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.14.mlp.fc1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.14.mlp.fc2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.14.mlp.fc2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.14.norm1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.14.norm1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.14.norm2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.14.norm2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.15.ls1
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.15.ls2
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.15.attn.qkv.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.15.attn.qkv.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.15.attn.proj.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.15.attn.proj.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.15.mlp.fc1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.15.mlp.fc1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.15.mlp.fc2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.15.mlp.fc2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.15.norm1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.15.norm1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.15.norm2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.15.norm2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.16.ls1
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.16.ls2
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.16.attn.qkv.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.16.attn.qkv.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.16.attn.proj.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.16.attn.proj.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.16.mlp.fc1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.16.mlp.fc1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.16.mlp.fc2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.16.mlp.fc2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.16.norm1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.16.norm1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.16.norm2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.16.norm2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.17.ls1
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.17.ls2
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.17.attn.qkv.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.17.attn.qkv.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.17.attn.proj.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.17.attn.proj.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.17.mlp.fc1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.17.mlp.fc1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.17.mlp.fc2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.17.mlp.fc2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.17.norm1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.17.norm1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.17.norm2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.17.norm2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.18.ls1
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.18.ls2
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.18.attn.qkv.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.18.attn.qkv.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.18.attn.proj.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.18.attn.proj.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.18.mlp.fc1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.18.mlp.fc1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.18.mlp.fc2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.18.mlp.fc2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.18.norm1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.18.norm1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.18.norm2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.18.norm2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.19.ls1
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.19.ls2
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.19.attn.qkv.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.19.attn.qkv.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.19.attn.proj.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.19.attn.proj.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.19.mlp.fc1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.19.mlp.fc1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.19.mlp.fc2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.19.mlp.fc2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.19.norm1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.19.norm1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.19.norm2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.19.norm2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.20.ls1
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.20.ls2
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.20.attn.qkv.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.20.attn.qkv.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.20.attn.proj.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.20.attn.proj.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.20.mlp.fc1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.20.mlp.fc1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.20.mlp.fc2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.20.mlp.fc2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.20.norm1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.20.norm1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.20.norm2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.20.norm2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.21.ls1
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.21.ls2
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.21.attn.qkv.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.21.attn.qkv.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.21.attn.proj.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.21.attn.proj.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.21.mlp.fc1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.21.mlp.fc1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.21.mlp.fc2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.21.mlp.fc2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.21.norm1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.21.norm1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.21.norm2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.21.norm2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.22.ls1
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.22.ls2
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.22.attn.qkv.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.22.attn.qkv.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.22.attn.proj.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.22.attn.proj.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.22.mlp.fc1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.22.mlp.fc1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.22.mlp.fc2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.22.mlp.fc2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.22.norm1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.22.norm1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.22.norm2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.22.norm2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.23.ls1
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.23.ls2
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.23.attn.qkv.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.23.attn.qkv.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.23.attn.proj.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.23.attn.proj.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.23.mlp.fc1.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.23.mlp.fc1.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.23.mlp.fc2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.23.mlp.fc2.bias
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.23.norm1.weight
Total parameters: ~7991.79 MB)02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.23.norm1.bias

Trainable parameters: ~7991.79 MB)
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.23.norm2.weight
02/24/2026 00:44:17 - INFO - __main__ - vision_model.encoder.layers.23.norm2.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.embed_tokens.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.0.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.0.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.0.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.0.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.0.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.0.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.0.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.0.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.0.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.0.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.0.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.0.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.1.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.1.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.1.self_attn.k_proj.weight
data_args.use_packed_ds False02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.1.self_attn.k_proj.bias

02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.1.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.1.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.1.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.1.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.1.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.1.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.1.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.1.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.2.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.2.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.2.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.2.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.2.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.2.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.2.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.2.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.2.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.2.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.2.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.2.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.3.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.3.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.3.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.3.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.3.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.3.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.3.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.3.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.3.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.3.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.3.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.3.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.4.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.4.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.4.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.4.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.4.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.4.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.4.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.4.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.4.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.4.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.4.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.4.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.5.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.5.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.5.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.5.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.5.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.5.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.5.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.5.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.5.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.5.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.5.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.5.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.6.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.6.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.6.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.6.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.6.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.6.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.6.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.6.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.6.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.6.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.6.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.6.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.7.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.7.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.7.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.7.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.7.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.7.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.7.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.7.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.7.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.7.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.7.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.7.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.8.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.8.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.8.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.8.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.8.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.8.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.8.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.8.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.8.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.8.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.8.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.8.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.9.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.9.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.9.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.9.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.9.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.9.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.9.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.9.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.9.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.9.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.9.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.9.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.10.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.10.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.10.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.10.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.10.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.10.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.10.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.10.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.10.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.10.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.10.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.10.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.11.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.11.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.11.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.11.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.11.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.11.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.11.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.11.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.11.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.11.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.11.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.11.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.12.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.12.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.12.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.12.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.12.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.12.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.12.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.12.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.12.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.12.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.12.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.12.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.13.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.13.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.13.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.13.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.13.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.13.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.13.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.13.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.13.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.13.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.13.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.13.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.14.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.14.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.14.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.14.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.14.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.14.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.14.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.14.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.14.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.14.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.14.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.14.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.15.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.15.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.15.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.15.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.15.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.15.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.15.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.15.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.15.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.15.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.15.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.15.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.16.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.16.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.16.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.16.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.16.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.16.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.16.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.16.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.16.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.16.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.16.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.16.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.17.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.17.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.17.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.17.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.17.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.17.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.17.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.17.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.17.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.17.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.17.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.17.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.18.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.18.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.18.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.18.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.18.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.18.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.18.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.18.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.18.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.18.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.18.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.18.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.19.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.19.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.19.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.19.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.19.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.19.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.19.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.19.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.19.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.19.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.19.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.19.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.20.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.20.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.20.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.20.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.20.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.20.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.20.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.20.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.20.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.20.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.20.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.20.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.21.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.21.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.21.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.21.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.21.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.21.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.21.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.21.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.21.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.21.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.21.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.21.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.22.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.22.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.22.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.22.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.22.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.22.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.22.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.22.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.22.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.22.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.22.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.22.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.23.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.23.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.23.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.23.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.23.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.23.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.23.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.23.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.23.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.23.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.23.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.23.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.24.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.24.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.24.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.24.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.24.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.24.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.24.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.24.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.24.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.24.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.24.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.24.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.25.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.25.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.25.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.25.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.25.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.25.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.25.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.25.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.25.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.25.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.25.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.25.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.26.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.26.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.26.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.26.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.26.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.26.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.26.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.26.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.26.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.26.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.26.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.26.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.27.self_attn.q_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.27.self_attn.q_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.27.self_attn.k_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.27.self_attn.k_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.27.self_attn.v_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.27.self_attn.v_proj.bias
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.27.self_attn.o_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.27.mlp.gate_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.27.mlp.up_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.27.mlp.down_proj.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.27.input_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.layers.27.post_attention_layernorm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.model.norm.weight
02/24/2026 00:44:17 - INFO - __main__ - language_model.lm_head.weight
02/24/2026 00:44:17 - INFO - __main__ - mlp1.0.weight
02/24/2026 00:44:17 - INFO - __main__ - mlp1.0.bias
02/24/2026 00:44:17 - INFO - __main__ - mlp1.1.weight
02/24/2026 00:44:17 - INFO - __main__ - mlp1.1.bias
02/24/2026 00:44:17 - INFO - __main__ - mlp1.3.weight
02/24/2026 00:44:17 - INFO - __main__ - mlp1.3.bias
02/24/2026 00:44:17 - INFO - __main__ - motion_mlp.0.weight
02/24/2026 00:44:17 - INFO - __main__ - motion_mlp.0.bias
02/24/2026 00:44:17 - INFO - __main__ - motion_mlp.1.weight
02/24/2026 00:44:17 - INFO - __main__ - motion_mlp.1.bias
02/24/2026 00:44:17 - INFO - __main__ - motion_mlp.3.weight
02/24/2026 00:44:17 - INFO - __main__ - motion_mlp.3.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.0.multipathway_blocks.0.conv.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.0.multipathway_blocks.0.norm.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.0.multipathway_blocks.0.norm.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.0.multipathway_blocks.1.conv.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.0.multipathway_blocks.1.norm.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.0.multipathway_blocks.1.norm.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.0.multipathway_fusion.conv_fast_to_slow.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.0.multipathway_fusion.norm.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.0.multipathway_fusion.norm.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.0.branch1_conv.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.0.branch1_norm.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.0.branch1_norm.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.0.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.0.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.0.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.0.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.0.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.0.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.0.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.0.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.0.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.1.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.1.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.1.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.1.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.1.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.1.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.1.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.1.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.1.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.2.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.2.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.2.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.2.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.2.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.2.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.2.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.2.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.0.res_blocks.2.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.0.branch1_conv.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.0.branch1_norm.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.0.branch1_norm.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.0.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.0.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.0.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.0.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.0.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.0.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.0.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.0.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.0.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.1.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.1.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.1.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.1.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.1.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.1.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.1.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.1.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.1.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.2.branch2.conv_a.weight

Loading checkpoint shards: 100%|██████████| 4/4 [01:03<00:00, 13.71s/it]02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.2.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.2.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.2.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.2.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.2.branch2.norm_b.bias

Loading checkpoint shards: 100%|██████████| 4/4 [01:03<00:00, 15.94s/it]
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.2.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.2.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_blocks.1.res_blocks.2.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_fusion.conv_fast_to_slow.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_fusion.norm.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.1.multipathway_fusion.norm.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.0.branch1_conv.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.0.branch1_norm.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.0.branch1_norm.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.0.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.0.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.0.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.0.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.0.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.0.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.0.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.0.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.0.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.1.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.1.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.1.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.1.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.1.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.1.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.1.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.1.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.1.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.2.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.2.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.2.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.2.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.2.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.2.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.2.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.2.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.2.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.3.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.3.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.3.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.3.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.3.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.3.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.3.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.3.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.0.res_blocks.3.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.0.branch1_conv.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.0.branch1_norm.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.0.branch1_norm.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.0.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.0.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.0.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.0.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.0.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.0.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.0.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.0.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.0.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.1.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.1.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.1.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.1.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.1.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.1.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.1.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.1.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.1.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.2.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.2.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.2.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.2.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.2.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.2.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.2.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.2.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.2.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.3.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.3.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.3.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.3.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.3.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.3.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.3.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.3.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_blocks.1.res_blocks.3.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_fusion.conv_fast_to_slow.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_fusion.norm.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.2.multipathway_fusion.norm.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.0.branch1_conv.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.0.branch1_norm.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.0.branch1_norm.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.0.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.0.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.0.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.0.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.0.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.0.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.0.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.0.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.0.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.1.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.1.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.1.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.1.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.1.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.1.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.1.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.1.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.1.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.2.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.2.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.2.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.2.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.2.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.2.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.2.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.2.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.2.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.3.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.3.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.3.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.3.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.3.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.3.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.3.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.3.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.3.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.4.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.4.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.4.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.4.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.4.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.4.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.4.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.4.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.4.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.5.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.5.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.5.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.5.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.5.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.5.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.5.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.5.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.0.res_blocks.5.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.0.branch1_conv.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.0.branch1_norm.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.0.branch1_norm.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.0.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.0.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.0.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.0.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.0.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.0.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.0.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.0.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.0.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.1.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.1.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.1.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.1.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.1.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.1.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.1.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.1.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.1.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.2.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.2.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.2.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.2.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.2.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.2.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.2.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.2.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.2.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.3.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.3.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.3.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.3.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.3.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.3.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.3.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.3.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.3.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.4.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.4.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.4.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.4.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.4.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.4.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.4.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.4.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.4.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.5.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.5.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.5.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.5.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.5.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.5.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.5.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.5.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_blocks.1.res_blocks.5.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_fusion.conv_fast_to_slow.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_fusion.norm.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.3.multipathway_fusion.norm.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.0.branch1_conv.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.0.branch1_norm.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.0.branch1_norm.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.0.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.0.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.0.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.0.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.0.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.0.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.0.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.0.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.0.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.1.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.1.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.1.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.1.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.1.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.1.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.1.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.1.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.1.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.2.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.2.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.2.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.2.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.2.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.2.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.2.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.2.branch2.norm_c.weight
Total parameters: ~7991.79 MB)
Trainable parameters: ~7991.79 MB)
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.0.res_blocks.2.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.0.branch1_conv.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.0.branch1_norm.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.0.branch1_norm.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.0.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.0.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.0.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.0.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.0.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.0.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.0.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.0.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.0.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.1.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.1.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.1.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.1.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.1.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.1.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.1.branch2.conv_c.weight
data_args.use_packed_ds False
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.1.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.1.branch2.norm_c.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.2.branch2.conv_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.2.branch2.norm_a.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.2.branch2.norm_a.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.2.branch2.conv_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.2.branch2.norm_b.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.2.branch2.norm_b.bias
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.2.branch2.conv_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.2.branch2.norm_c.weight
02/24/2026 00:44:17 - INFO - __main__ - slowfast_model.feature_extraction.4.multipathway_blocks.1.res_blocks.2.branch2.norm_c.bias
Total parameters: ~7991.79 MB)
Trainable parameters: ~7991.79 MB)
data_args.use_packed_ds False
Total parameters: ~7991.79 MB)
Trainable parameters: ~7991.79 MB)
data_args.use_packed_ds False

Loading checkpoint shards: 100%|██████████| 4/4 [01:03<00:00, 13.72s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [01:03<00:00, 15.96s/it]

Loading checkpoint shards: 100%|██████████| 4/4 [01:04<00:00, 13.75s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [01:04<00:00, 16.00s/it]
Total parameters: ~7991.79 MB)
Trainable parameters: ~7991.79 MB)
data_args.use_packed_ds False
Total parameters: ~7991.79 MB)
Trainable parameters: ~7991.79 MB)
data_args.use_packed_ds False
[INFO|trainer.py:571] 2026-02-24 00:44:17,495 >> Using auto half precision backend
[2026-02-24 00:44:17,590] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed info: version=0.15.4, git-hash=unknown, git-branch=unknown
[2026-02-24 00:44:17,590] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
[2026-02-24 00:44:29,070] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
Using /root/.cache/torch_extensions/py311_cu124 as PyTorch extensions root...
Using /root/.cache/torch_extensions/py311_cu124 as PyTorch extensions root...
Using /root/.cache/torch_extensions/py311_cu124 as PyTorch extensions root...
Using /root/.cache/torch_extensions/py311_cu124 as PyTorch extensions root...
Using /root/.cache/torch_extensions/py311_cu124 as PyTorch extensions root...
Using /root/.cache/torch_extensions/py311_cu124 as PyTorch extensions root...
Using /root/.cache/torch_extensions/py311_cu124 as PyTorch extensions root...
Using /root/.cache/torch_extensions/py311_cu124 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /root/.cache/torch_extensions/py311_cu124/fused_adam/build.ninja...
Building extension module fused_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module fused_adam...
Time to load fused_adam op: 1.3796133995056152 seconds
Loading extension module fused_adam...Loading extension module fused_adam...

Loading extension module fused_adam...
Loading extension module fused_adam...
Loading extension module fused_adam...
Loading extension module fused_adam...
Loading extension module fused_adam...
Time to load fused_adam op: 1.4109270572662354 secondsTime to load fused_adam op: 1.410923957824707 seconds

Time to load fused_adam op: 1.410433053970337 seconds
Time to load fused_adam op: 1.4112443923950195 seconds
Time to load fused_adam op: 1.4099907875061035 seconds
Time to load fused_adam op: 1.411553144454956 seconds
Time to load fused_adam op: 1.4110934734344482 seconds
[2026-02-24 00:44:30,483] [INFO] [logging.py:128:log_dist] [Rank 0] Using DeepSpeed Optimizer param name adamw as basic optimizer
[2026-02-24 00:44:30,483] [INFO] [logging.py:128:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
[2026-02-24 00:44:30,546] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Basic Optimizer = FusedAdam
[2026-02-24 00:44:30,547] [INFO] [utils.py:59:is_zero_supported_optimizer] Checking ZeRO support for optimizer=FusedAdam type=<class 'deepspeed.ops.adam.fused_adam.FusedAdam'>
[2026-02-24 00:44:30,547] [INFO] [logging.py:128:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 1 optimizer
[2026-02-24 00:44:30,547] [INFO] [stage_1_and_2.py:149:__init__] Reduce bucket size 1000000000
[2026-02-24 00:44:30,547] [INFO] [stage_1_and_2.py:150:__init__] Allgather bucket size 1000000000
[2026-02-24 00:44:30,547] [INFO] [stage_1_and_2.py:151:__init__] CPU Offload: False
[2026-02-24 00:44:30,547] [INFO] [stage_1_and_2.py:152:__init__] Round robin gradient partitioning: False
[2026-02-24 00:44:50,262] [INFO] [utils.py:781:see_memory_usage] Before initializing optimizer states
[2026-02-24 00:44:50,326] [INFO] [utils.py:782:see_memory_usage] MA 19.05 GB         Max_MA 20.91 GB         CA 21.12 GB         Max_CA 21 GB 
[2026-02-24 00:44:50,328] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 67.8 GB, percent = 3.9%
[2026-02-24 00:44:50,415] [INFO] [utils.py:781:see_memory_usage] After initializing optimizer states
[2026-02-24 00:44:50,415] [INFO] [utils.py:782:see_memory_usage] MA 19.05 GB         Max_MA 22.77 GB         CA 24.85 GB         Max_CA 25 GB 
[2026-02-24 00:44:50,417] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 67.8 GB, percent = 3.9%
[2026-02-24 00:44:50,417] [INFO] [stage_1_and_2.py:544:__init__] optimizer state initialized
[2026-02-24 00:44:50,495] [INFO] [utils.py:781:see_memory_usage] After initializing ZeRO optimizer
[2026-02-24 00:44:50,496] [INFO] [utils.py:782:see_memory_usage] MA 19.05 GB         Max_MA 19.05 GB         CA 24.85 GB         Max_CA 25 GB 
[2026-02-24 00:44:50,497] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 67.8 GB, percent = 3.9%
[2026-02-24 00:44:50,501] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Final Optimizer = DeepSpeedZeroOptimizer
[2026-02-24 00:44:50,501] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed using client callable to create LR scheduler
[2026-02-24 00:44:50,501] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed LR Scheduler = <torch.optim.lr_scheduler.LambdaLR object at 0x7fc6ec253b50>
[2026-02-24 00:44:50,501] [INFO] [logging.py:128:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0], mom=[[0.9, 0.999]]
[2026-02-24 00:44:50,503] [INFO] [config.py:999:print] DeepSpeedEngine configuration:
[2026-02-24 00:44:50,503] [INFO] [config.py:1003:print]   activation_checkpointing_config  {
    "partition_activations": false, 
    "contiguous_memory_optimization": false, 
    "cpu_checkpointing": false, 
    "number_checkpoints": null, 
    "synchronize_checkpoint_boundary": false, 
    "profile": false
}
[2026-02-24 00:44:50,503] [INFO] [config.py:1003:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True, 'use_gds': False}
[2026-02-24 00:44:50,503] [INFO] [config.py:1003:print]   amp_enabled .................. False
[2026-02-24 00:44:50,503] [INFO] [config.py:1003:print]   amp_params ................... False
[2026-02-24 00:44:50,503] [INFO] [config.py:1003:print]   autotuning_config ............ {
    "enabled": false, 
    "start_step": null, 
    "end_step": null, 
    "metric_path": null, 
    "arg_mappings": null, 
    "metric": "throughput", 
    "model_info": null, 
    "results_dir": "autotuning_results", 
    "exps_dir": "autotuning_exps", 
    "overwrite": true, 
    "fast": true, 
    "start_profile_step": 3, 
    "end_profile_step": 5, 
    "tuner_type": "gridsearch", 
    "tuner_early_stopping": 5, 
    "tuner_num_trials": 50, 
    "model_info_path": null, 
    "mp_size": 1, 
    "max_train_batch_size": null, 
    "min_train_batch_size": 1, 
    "max_train_micro_batch_size_per_gpu": 1.024000e+03, 
    "min_train_micro_batch_size_per_gpu": 1, 
    "num_tuning_micro_batch_sizes": 3
}
[2026-02-24 00:44:50,503] [INFO] [config.py:1003:print]   bfloat16_enabled ............. True
[2026-02-24 00:44:50,503] [INFO] [config.py:1003:print]   bfloat16_immediate_grad_update  False
[2026-02-24 00:44:50,503] [INFO] [config.py:1003:print]   checkpoint_parallel_write_pipeline  False
[2026-02-24 00:44:50,503] [INFO] [config.py:1003:print]   checkpoint_tag_validation_enabled  True
[2026-02-24 00:44:50,503] [INFO] [config.py:1003:print]   checkpoint_tag_validation_fail  False
[2026-02-24 00:44:50,503] [INFO] [config.py:1003:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7fc70d47a550>
[2026-02-24 00:44:50,503] [INFO] [config.py:1003:print]   communication_data_type ...... None
[2026-02-24 00:44:50,503] [INFO] [config.py:1003:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2026-02-24 00:44:50,503] [INFO] [config.py:1003:print]   curriculum_enabled_legacy .... False
[2026-02-24 00:44:50,503] [INFO] [config.py:1003:print]   curriculum_params_legacy ..... False
[2026-02-24 00:44:50,503] [INFO] [config.py:1003:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2026-02-24 00:44:50,503] [INFO] [config.py:1003:print]   data_efficiency_enabled ...... False
[2026-02-24 00:44:50,503] [INFO] [config.py:1003:print]   dataloader_drop_last ......... False
[2026-02-24 00:44:50,503] [INFO] [config.py:1003:print]   disable_allgather ............ False
[2026-02-24 00:44:50,503] [INFO] [config.py:1003:print]   dump_state ................... False
[2026-02-24 00:44:50,503] [INFO] [config.py:1003:print]   dynamic_loss_scale_args ...... None
[2026-02-24 00:44:50,503] [INFO] [config.py:1003:print]   eigenvalue_enabled ........... False
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   eigenvalue_gas_boundary_resolution  1
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   eigenvalue_layer_num ......... 0
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   eigenvalue_max_iter .......... 100
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   eigenvalue_stability ......... 1e-06
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   eigenvalue_tol ............... 0.01
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   eigenvalue_verbose ........... False
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   elasticity_enabled ........... False
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   flops_profiler_config ........ {
    "enabled": false, 
    "recompute_fwd_factor": 0.0, 
    "profile_step": 1, 
    "module_depth": -1, 
    "top_modules": 1, 
    "detailed": true, 
    "output_file": null
}
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   fp16_auto_cast ............... None
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   fp16_enabled ................. False
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   fp16_master_weights_and_gradients  False
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   global_rank .................. 0
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   grad_accum_dtype ............. None
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   gradient_accumulation_steps .. 2
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   gradient_clipping ............ 1.0
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   gradient_predivide_factor .... 1.0
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   graph_harvesting ............. False
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   initial_dynamic_scale ........ 1
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   load_universal_checkpoint .... False
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   loss_scale ................... 1.0
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   memory_breakdown ............. False
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   mics_hierarchial_params_gather  False
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   mics_shard_size .............. -1
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') comet=CometConfig(enabled=False, samples_log_interval=100, project=None, workspace=None, api_key=None, experiment_name=None, experiment_key=None, online=None, mode=None) wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName')
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   nebula_config ................ {
    "enabled": false, 
    "persistent_storage_path": null, 
    "persistent_time_interval": 100, 
    "num_of_version_in_retention": 2, 
    "enable_nebula_load": true, 
    "load_path": null
}
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   optimizer_legacy_fusion ...... False
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   optimizer_name ............... adamw
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   optimizer_params ............. {'lr': 2e-05, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.05}
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   pld_enabled .................. False
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   pld_params ................... False
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   prescale_gradients ........... False
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   scheduler_name ............... None
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   scheduler_params ............. None
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   seq_parallel_communication_data_type  torch.float32
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   sparse_attention ............. None
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   sparse_gradients_enabled ..... False
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   steps_per_print .............. inf
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   timers_config ................ enabled=True synchronized=True
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   train_batch_size ............. 16
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   train_micro_batch_size_per_gpu  1
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   use_data_before_expert_parallel_  False
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   use_node_local_storage ....... False
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   wall_clock_breakdown ......... True
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   weight_quantization_config ... None
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   world_size ................... 8
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   zero_allow_untested_optimizer  False
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   zero_config .................. stage=1 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=1000000000 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=1000000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50000000 param_persistence_threshold=100000 model_persistence_threshold=9223372036854775807 max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=False use_all_reduce_for_fetch_params=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   zero_enabled ................. True
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   zero_force_ds_cpu_optimizer .. True
[2026-02-24 00:44:50,504] [INFO] [config.py:1003:print]   zero_optimization_stage ...... 1
[2026-02-24 00:44:50,504] [INFO] [config.py:989:print_user_config]   json = {
    "zero_optimization": {
        "stage": 1, 
        "allgather_partitions": true, 
        "allgather_bucket_size": 1.000000e+09, 
        "overlap_comm": true, 
        "reduce_scatter": true, 
        "reduce_bucket_size": 1.000000e+09, 
        "contiguous_gradients": true
    }, 
    "fp16": {
        "enabled": false, 
        "auto_cast": true, 
        "loss_scale": 0, 
        "initial_scale_power": 32, 
        "loss_scale_window": 1000, 
        "hysteresis": 2, 
        "min_loss_scale": 1
    }, 
    "bf16": {
        "enabled": true
    }, 
    "optimizer": {
        "type": "AdamW", 
        "params": {
            "lr": 2e-05, 
            "betas": [0.9, 0.999], 
            "eps": 1e-08, 
            "weight_decay": 0.05
        }
    }, 
    "gradient_accumulation_steps": 2, 
    "gradient_clipping": 1.0, 
    "steps_per_print": inf, 
    "train_batch_size": 16, 
    "train_micro_batch_size_per_gpu": 1, 
    "wall_clock_breakdown": true
}
[INFO|trainer.py:1721] 2026-02-24 00:44:50,504 >> ***** Running training *****
[INFO|trainer.py:1722] 2026-02-24 00:44:50,505 >>   Num examples = 478
[INFO|trainer.py:1723] 2026-02-24 00:44:50,505 >>   Num Epochs = 3
[INFO|trainer.py:1724] 2026-02-24 00:44:50,505 >>   Instantaneous batch size per device = 1
[INFO|trainer.py:1727] 2026-02-24 00:44:50,505 >>   Total train batch size (w. parallel, distributed & accumulation) = 16
[INFO|trainer.py:1728] 2026-02-24 00:44:50,505 >>   Gradient Accumulation steps = 2
[INFO|trainer.py:1729] 2026-02-24 00:44:50,505 >>   Total optimization steps = 90
[INFO|trainer.py:1730] 2026-02-24 00:44:50,506 >>   Number of trainable parameters = 7,991,788,488

  0%|          | 0/90 [00:00<?, ?it/s][2026-02-24 00:44:56,091] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:44:56,102] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:44:56,112] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:44:56,118] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:44:56,122] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:44:56,130] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:44:56,130] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:44:56,130] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:45:03,504] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:45:03,524] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:45:03,530] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:45:03,533] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:45:03,539] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:45:03,540] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:45:03,544] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:45:03,546] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:45:08,480] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:45:08,547] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:45:08,610] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:45:08,634] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:45:08,652] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:45:08,664] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:45:08,677] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:45:08,678] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:45:13,919] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:45:13,935] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:45:13,940] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:45:13,945] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:45:13,966] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:45:13,977] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:45:13,997] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:45:13,998] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:45:30,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 911.40 | bwd_microstep: 1255.80 | bwd_inner_microstep: 1255.58 | bwd_allreduce_microstep: 0.07 | step_microstep: 0.05
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:45:45,302] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 96.14 | optimizer_gradients: 9.86 | optimizer_step: 15.50
[2026-02-24 00:45:45,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 335.40 | bwd_microstep: 8184.41 | bwd_inner_microstep: 8144.51 | bwd_allreduce_microstep: 39.86 | step_microstep: 523.81
[2026-02-24 00:45:45,303] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 1246.81 | bwd: 9440.21 | bwd_inner: 9400.12 | bwd_allreduce: 39.93 | step: 523.87

  1%|          | 1/90 [00:54<1:21:16, 54.79s/it]
                                                
{'loss': 3.749, 'learning_rate': 6.666666666666667e-06, 'epoch': 0.03}

  1%|          | 1/90 [00:55<1:21:16, 54.79s/it]dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2979
[2026-02-24 00:45:47,391] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 542.04 | bwd_microstep: 694.76 | bwd_inner_microstep: 694.74 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.01
dynamic ViT batch size: 536, images per sample: 536.0, dynamic token length: 2711
[2026-02-24 00:45:50,410] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 60.04 | optimizer_gradients: 9.55 | optimizer_step: 9.91
[2026-02-24 00:45:50,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 531.49 | bwd_microstep: 2352.70 | bwd_inner_microstep: 2328.97 | bwd_allreduce_microstep: 23.64 | step_microstep: 101.51
[2026-02-24 00:45:50,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 1073.53 | bwd: 3047.47 | bwd_inner: 3023.74 | bwd_allreduce: 23.65 | step: 101.52

  2%|▏         | 2/90 [00:59<37:29, 25.56s/it]  
                                              
{'loss': 3.6556, 'learning_rate': 1.3333333333333333e-05, 'epoch': 0.07}

  2%|▏         | 2/90 [00:59<37:29, 25.56s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:45:51,267] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.52 | bwd_microstep: 503.56 | bwd_inner_microstep: 503.54 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.04
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:45:53,065] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.40 | optimizer_gradients: 9.61 | optimizer_step: 9.89
[2026-02-24 00:45:53,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.74 | bwd_microstep: 1365.81 | bwd_inner_microstep: 1324.90 | bwd_allreduce_microstep: 40.87 | step_microstep: 90.45
[2026-02-24 00:45:53,066] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 647.27 | bwd: 1869.38 | bwd_inner: 1828.44 | bwd_allreduce: 40.89 | step: 90.50

  3%|▎         | 3/90 [01:02<21:54, 15.10s/it]
                                              
{'loss': 1.1387, 'learning_rate': 2e-05, 'epoch': 0.1}

  3%|▎         | 3/90 [01:02<21:54, 15.10s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:45:53,923] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.15 | bwd_microstep: 503.26 | bwd_inner_microstep: 503.23 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.06
dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2979
[h264 @ 0x960d980] mmco: unref short failure
[h264 @ 0x960d980] mmco: unref short failure
[2026-02-24 00:45:55,799] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.28 | optimizer_gradients: 9.57 | optimizer_step: 9.91
[2026-02-24 00:45:55,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 545.59 | bwd_microstep: 1209.93 | bwd_inner_microstep: 1187.22 | bwd_allreduce_microstep: 22.67 | step_microstep: 90.85
[2026-02-24 00:45:55,801] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 869.74 | bwd: 1713.20 | bwd_inner: 1690.45 | bwd_allreduce: 22.69 | step: 90.92

  4%|▍         | 4/90 [01:05<14:38, 10.22s/it]
                                              
{'loss': 0.4948, 'learning_rate': 1.999348095389677e-05, 'epoch': 0.13}

  4%|▍         | 4/90 [01:05<14:38, 10.22s/it]dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2715
[2026-02-24 00:45:57,044] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 535.10 | bwd_microstep: 663.41 | bwd_inner_microstep: 663.38 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.12
dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2979
[2026-02-24 00:45:58,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 59.34 | optimizer_gradients: 9.55 | optimizer_step: 9.92
[2026-02-24 00:45:58,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 544.47 | bwd_microstep: 1025.46 | bwd_inner_microstep: 1010.63 | bwd_allreduce_microstep: 14.79 | step_microstep: 92.81
[2026-02-24 00:45:58,739] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 1079.56 | bwd: 1688.87 | bwd_inner: 1674.01 | bwd_allreduce: 14.81 | step: 92.96

  6%|▌         | 5/90 [01:08<10:45,  7.59s/it]
                                              
{'loss': 0.4703, 'learning_rate': 1.9973932315179502e-05, 'epoch': 0.17}

  6%|▌         | 5/90 [01:08<10:45,  7.59s/it]dynamic ViT batch size: 244, images per sample: 244.0, dynamic token length: 2419
[2026-02-24 00:45:59,522] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.59 | bwd_microstep: 459.17 | bwd_inner_microstep: 459.14 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.05
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:46:04,272] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.06 | optimizer_gradients: 9.57 | optimizer_step: 9.89
[2026-02-24 00:46:04,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.62 | bwd_microstep: 4315.56 | bwd_inner_microstep: 4082.93 | bwd_allreduce_microstep: 232.58 | step_microstep: 91.04
[2026-02-24 00:46:04,273] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 616.21 | bwd: 4774.74 | bwd_inner: 4542.07 | bwd_allreduce: 232.60 | step: 91.10

  7%|▋         | 6/90 [01:13<09:39,  6.89s/it]
                                              
{'loss': 0.155, 'learning_rate': 1.9941379571543597e-05, 'epoch': 0.2}

  7%|▋         | 6/90 [01:13<09:39,  6.89s/it]dynamic ViT batch size: 264, images per sample: 264.0, dynamic token length: 2439
[2026-02-24 00:46:05,079] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 307.84 | bwd_microstep: 472.53 | bwd_inner_microstep: 472.51 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.05
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:46:10,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.50 | optimizer_gradients: 9.56 | optimizer_step: 9.88
[2026-02-24 00:46:10,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.11 | bwd_microstep: 3629.55 | bwd_inner_microstep: 3584.88 | bwd_allreduce_microstep: 44.63 | step_microstep: 90.32
[2026-02-24 00:46:10,250] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 631.95 | bwd: 4102.09 | bwd_inner: 4057.39 | bwd_allreduce: 44.65 | step: 90.38

  8%|▊         | 7/90 [01:19<09:07,  6.59s/it]
                                              
{'loss': 0.2287, 'learning_rate': 1.9895865165556375e-05, 'epoch': 0.23}

  8%|▊         | 7/90 [01:19<09:07,  6.59s/it]dynamic ViT batch size: 544, images per sample: 544.0, dynamic token length: 2983
[2026-02-24 00:46:11,534] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 546.30 | bwd_microstep: 697.85 | bwd_inner_microstep: 697.82 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.05
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:46:12,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.27 | optimizer_gradients: 9.60 | optimizer_step: 9.88
[2026-02-24 00:46:12,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.85 | bwd_microstep: 1030.15 | bwd_inner_microstep: 986.23 | bwd_allreduce_microstep: 43.87 | step_microstep: 90.34
[2026-02-24 00:46:12,996] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 869.15 | bwd: 1728.00 | bwd_inner: 1684.06 | bwd_allreduce: 43.88 | step: 90.42

  9%|▉         | 8/90 [01:22<07:20,  5.37s/it]
                                              
{'loss': 0.1558, 'learning_rate': 1.9837448439320027e-05, 'epoch': 0.27}

  9%|▉         | 8/90 [01:22<07:20,  5.37s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:46:13,846] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.36 | bwd_microstep: 503.03 | bwd_inner_microstep: 503.01 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.09
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:46:20,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.61 | optimizer_gradients: 9.57 | optimizer_step: 9.88
[2026-02-24 00:46:20,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.53 | bwd_microstep: 589.14 | bwd_inner_microstep: 572.83 | bwd_allreduce_microstep: 16.28 | step_microstep: 90.97
[2026-02-24 00:46:20,255] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 644.88 | bwd: 1092.18 | bwd_inner: 1075.83 | bwd_allreduce: 16.30 | step: 91.08

 10%|█         | 9/90 [01:29<08:02,  5.96s/it]
                                              
{'loss': 0.1309, 'learning_rate': 1.976620555710087e-05, 'epoch': 0.3}

 10%|█         | 9/90 [01:29<08:02,  5.96s/it]dynamic ViT batch size: 544, images per sample: 544.0, dynamic token length: 2983
[2026-02-24 00:46:21,537] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 546.22 | bwd_microstep: 693.90 | bwd_inner_microstep: 693.88 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.19
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:46:24,737] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.49 | optimizer_gradients: 9.64 | optimizer_step: 9.88
[2026-02-24 00:46:24,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 328.07 | bwd_microstep: 2757.08 | bwd_inner_microstep: 2715.81 | bwd_allreduce_microstep: 41.23 | step_microstep: 91.94
[2026-02-24 00:46:24,738] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 874.28 | bwd: 3450.99 | bwd_inner: 3409.69 | bwd_allreduce: 41.24 | step: 92.22

 11%|█         | 10/90 [01:34<07:20,  5.50s/it]
                                               
{'loss': 0.1989, 'learning_rate': 1.9682229406025635e-05, 'epoch': 0.33}

 11%|█         | 10/90 [01:34<07:20,  5.50s/it]dynamic ViT batch size: 276, images per sample: 276.0, dynamic token length: 2715
[2026-02-24 00:46:25,612] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 331.47 | bwd_microstep: 516.79 | bwd_inner_microstep: 516.77 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.10
dynamic ViT batch size: 260, images per sample: 260.0, dynamic token length: 2435
[2026-02-24 00:46:34,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.23 | optimizer_gradients: 9.55 | optimizer_step: 9.88
[2026-02-24 00:46:34,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.54 | bwd_microstep: 794.53 | bwd_inner_microstep: 751.29 | bwd_allreduce_microstep: 43.19 | step_microstep: 89.92
[2026-02-24 00:46:34,058] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 636.01 | bwd: 1311.32 | bwd_inner: 1268.06 | bwd_allreduce: 43.20 | step: 90.03

 12%|█▏        | 11/90 [01:43<08:47,  6.67s/it]
                                               
{'loss': 0.1272, 'learning_rate': 1.9585629474974413e-05, 'epoch': 0.37}

 12%|█▏        | 11/90 [01:43<08:47,  6.67s/it]dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2979
[2026-02-24 00:46:35,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 542.98 | bwd_microstep: 689.79 | bwd_inner_microstep: 689.77 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.05
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:46:37,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.33 | optimizer_gradients: 9.62 | optimizer_step: 9.86
[2026-02-24 00:46:37,173] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.24 | bwd_microstep: 1413.18 | bwd_inner_microstep: 1369.19 | bwd_allreduce_microstep: 43.94 | step_microstep: 90.25
[2026-02-24 00:46:37,174] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 867.22 | bwd: 2102.98 | bwd_inner: 2058.96 | bwd_allreduce: 43.96 | step: 90.31

 13%|█▎        | 12/90 [01:46<07:16,  5.59s/it]
                                               
{'loss': 0.9614, 'learning_rate': 1.9476531711828027e-05, 'epoch': 0.4}

 13%|█▎        | 12/90 [01:46<07:16,  5.59s/it]dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2715
[2026-02-24 00:46:38,411] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 534.49 | bwd_microstep: 664.10 | bwd_inner_microstep: 664.08 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.07
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:46:45,893] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.49 | optimizer_gradients: 9.57 | optimizer_step: 9.87
[2026-02-24 00:46:45,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.29 | bwd_microstep: 7049.99 | bwd_inner_microstep: 7026.47 | bwd_allreduce_microstep: 23.48 | step_microstep: 90.23
[2026-02-24 00:46:45,894] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 857.78 | bwd: 7714.10 | bwd_inner: 7690.55 | bwd_allreduce: 23.49 | step: 90.31

 14%|█▍        | 13/90 [01:55<08:23,  6.54s/it]
                                               
{'loss': 0.4291, 'learning_rate': 1.935507835925601e-05, 'epoch': 0.43}

 14%|█▍        | 13/90 [01:55<08:23,  6.54s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:46:46,745] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.24 | bwd_microstep: 501.64 | bwd_inner_microstep: 501.61 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.04
dynamic ViT batch size: 248, images per sample: 248.0, dynamic token length: 2423
[2026-02-24 00:46:49,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 59.19 | optimizer_gradients: 9.56 | optimizer_step: 9.87
[2026-02-24 00:46:49,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.12 | bwd_microstep: 2397.65 | bwd_inner_microstep: 2353.09 | bwd_allreduce_microstep: 44.52 | step_microstep: 91.85
[2026-02-24 00:46:49,546] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 617.36 | bwd: 2899.29 | bwd_inner: 2854.71 | bwd_allreduce: 44.53 | step: 91.90

 16%|█▌        | 14/90 [01:59<07:10,  5.67s/it]
                                               
{'loss': 0.2756, 'learning_rate': 1.9221427769259333e-05, 'epoch': 0.47}

 16%|█▌        | 14/90 [01:59<07:10,  5.67s/it]dynamic ViT batch size: 280, images per sample: 280.0, dynamic token length: 2719
[2026-02-24 00:46:50,433] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 333.44 | bwd_microstep: 523.84 | bwd_inner_microstep: 523.81 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.08
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:46:52,210] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.95 | optimizer_gradients: 10.28 | optimizer_step: 9.89
[2026-02-24 00:46:52,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.90 | bwd_microstep: 1340.46 | bwd_inner_microstep: 1298.82 | bwd_allreduce_microstep: 41.59 | step_microstep: 94.79
[2026-02-24 00:46:52,211] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 657.33 | bwd: 1864.31 | bwd_inner: 1822.63 | bwd_allreduce: 41.62 | step: 94.88

 17%|█▋        | 15/90 [02:01<05:57,  4.76s/it]
                                               
{'loss': 0.2839, 'learning_rate': 1.9075754196709574e-05, 'epoch': 0.5}

 17%|█▋        | 15/90 [02:01<05:57,  4.76s/it]dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2715
[2026-02-24 00:46:53,447] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 533.86 | bwd_microstep: 664.18 | bwd_inner_microstep: 664.16 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.04
dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2715
[2026-02-24 00:46:54,889] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.47 | optimizer_gradients: 9.57 | optimizer_step: 9.89
[2026-02-24 00:46:54,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 533.85 | bwd_microstep: 787.65 | bwd_inner_microstep: 761.79 | bwd_allreduce_microstep: 25.83 | step_microstep: 90.10
[2026-02-24 00:46:54,890] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 1067.71 | bwd: 1451.84 | bwd_inner: 1425.94 | bwd_allreduce: 25.85 | step: 90.15

 18%|█▊        | 16/90 [02:04<05:05,  4.13s/it]
                                               
{'loss': 0.226, 'learning_rate': 1.8918247572153822e-05, 'epoch': 0.53}

 18%|█▊        | 16/90 [02:04<05:05,  4.13s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:46:55,751] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 325.04 | bwd_microstep: 503.98 | bwd_inner_microstep: 503.95 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.06
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:46:59,372] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.22 | optimizer_gradients: 9.67 | optimizer_step: 9.87
[2026-02-24 00:46:59,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.57 | bwd_microstep: 3187.72 | bwd_inner_microstep: 3161.29 | bwd_allreduce_microstep: 26.33 | step_microstep: 90.44
[2026-02-24 00:46:59,373] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 649.58 | bwd: 3691.70 | bwd_inner: 3665.29 | bwd_allreduce: 26.34 | step: 90.52

 19%|█▉        | 17/90 [02:08<05:09,  4.24s/it]
                                               
{'loss': 0.2706, 'learning_rate': 1.8749113254181498e-05, 'epoch': 0.57}

 19%|█▉        | 17/90 [02:08<05:09,  4.24s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[h264 @ 0x9efd140] mmco: unref short failure
[h264 @ 0x9efd140] mmco: unref short failure
[2026-02-24 00:47:01,549] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.26 | bwd_microstep: 501.87 | bwd_inner_microstep: 501.85 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.07
dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2979
[2026-02-24 00:47:03,003] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 59.30 | optimizer_gradients: 9.65 | optimizer_step: 9.88
[2026-02-24 00:47:03,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 543.70 | bwd_microstep: 788.15 | bwd_inner_microstep: 759.30 | bwd_allreduce_microstep: 28.82 | step_microstep: 91.83
[2026-02-24 00:47:03,004] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 867.95 | bwd: 1290.03 | bwd_inner: 1261.15 | bwd_allreduce: 28.83 | step: 91.92

 20%|██        | 18/90 [02:12<04:52,  4.06s/it]
                                               
{'loss': 0.7145, 'learning_rate': 1.8568571761675893e-05, 'epoch': 0.6}

 20%|██        | 18/90 [02:12<04:52,  4.06s/it]dynamic ViT batch size: 288, images per sample: 288.0, dynamic token length: 2727
[2026-02-24 00:47:03,900] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 341.29 | bwd_microstep: 526.30 | bwd_inner_microstep: 526.28 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.17
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:47:12,813] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.64 | optimizer_gradients: 9.61 | optimizer_step: 9.91
[2026-02-24 00:47:12,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.75 | bwd_microstep: 8480.56 | bwd_inner_microstep: 8439.42 | bwd_allreduce_microstep: 41.07 | step_microstep: 90.55
[2026-02-24 00:47:12,814] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 665.04 | bwd: 9006.87 | bwd_inner: 8965.72 | bwd_allreduce: 41.08 | step: 90.75

 21%|██        | 19/90 [02:22<06:50,  5.78s/it]
                                               
{'loss': 0.269, 'learning_rate': 1.837685848629965e-05, 'epoch': 0.63}

 21%|██        | 19/90 [02:22<06:50,  5.78s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:47:13,668] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.85 | bwd_microstep: 503.78 | bwd_inner_microstep: 503.75 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.05
dynamic ViT batch size: 288, images per sample: 288.0, dynamic token length: 2727
[2026-02-24 00:47:15,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.06 | optimizer_gradients: 9.67 | optimizer_step: 9.88
[2026-02-24 00:47:15,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 339.51 | bwd_microstep: 1393.90 | bwd_inner_microstep: 1350.00 | bwd_allreduce_microstep: 43.86 | step_microstep: 90.29
[2026-02-24 00:47:15,511] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 664.35 | bwd: 1897.68 | bwd_inner: 1853.75 | bwd_allreduce: 43.88 | step: 90.35

 22%|██▏       | 20/90 [02:24<05:40,  4.86s/it]
                                               
{'loss': 0.2044, 'learning_rate': 1.817422338558892e-05, 'epoch': 0.67}

 22%|██▏       | 20/90 [02:25<05:40,  4.86s/it]dynamic ViT batch size: 272, images per sample: 272.0, dynamic token length: 2711
[2026-02-24 00:47:16,380] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 327.62 | bwd_microstep: 513.86 | bwd_inner_microstep: 513.83 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.06
dynamic ViT batch size: 272, images per sample: 272.0, dynamic token length: 2711
[2026-02-24 00:47:17,971] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.17 | optimizer_gradients: 9.63 | optimizer_step: 9.91
[2026-02-24 00:47:17,972] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 326.57 | bwd_microstep: 1154.54 | bwd_inner_microstep: 1114.02 | bwd_allreduce_microstep: 40.45 | step_microstep: 90.76
[2026-02-24 00:47:17,973] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 654.19 | bwd: 1668.41 | bwd_inner: 1627.87 | bwd_allreduce: 40.46 | step: 90.84

 23%|██▎       | 21/90 [02:27<04:45,  4.14s/it]
                                               
{'loss': 0.1411, 'learning_rate': 1.796093065705644e-05, 'epoch': 0.7}

 23%|██▎       | 21/90 [02:27<04:45,  4.14s/it]dynamic ViT batch size: 544, images per sample: 544.0, dynamic token length: 2983
[2026-02-24 00:47:19,254] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 547.69 | bwd_microstep: 695.01 | bwd_inner_microstep: 694.98 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.19
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:47:20,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.32 | optimizer_gradients: 9.69 | optimizer_step: 9.91
[2026-02-24 00:47:20,464] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.41 | bwd_microstep: 774.69 | bwd_inner_microstep: 731.60 | bwd_allreduce_microstep: 43.05 | step_microstep: 90.61
[2026-02-24 00:47:20,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 872.10 | bwd: 1469.71 | bwd_inner: 1426.59 | bwd_allreduce: 43.07 | step: 90.81

 24%|██▍       | 22/90 [02:29<04:07,  3.64s/it]
                                               
{'loss': 0.1101, 'learning_rate': 1.7737258393728363e-05, 'epoch': 0.73}

 24%|██▍       | 22/90 [02:29<04:07,  3.64s/it]dynamic ViT batch size: 272, images per sample: 272.0, dynamic token length: 2711
[2026-02-24 00:47:21,326] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 327.08 | bwd_microstep: 506.26 | bwd_inner_microstep: 506.23 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.08
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:47:25,184] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.39 | optimizer_gradients: 9.66 | optimizer_step: 9.92
[2026-02-24 00:47:25,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.84 | bwd_microstep: 773.18 | bwd_inner_microstep: 729.16 | bwd_allreduce_microstep: 43.98 | step_microstep: 90.74
[2026-02-24 00:47:25,185] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 649.92 | bwd: 1279.45 | bwd_inner: 1235.40 | bwd_allreduce: 43.99 | step: 90.83

 26%|██▌       | 23/90 [02:34<04:25,  3.97s/it]
                                               
{'loss': 0.1022, 'learning_rate': 1.7503498221564026e-05, 'epoch': 0.77}

 26%|██▌       | 23/90 [02:34<04:25,  3.97s/it]dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2979
[2026-02-24 00:47:26,465] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 543.93 | bwd_microstep: 690.28 | bwd_inner_microstep: 690.25 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.06
dynamic ViT batch size: 544, images per sample: 544.0, dynamic token length: 2983
[2026-02-24 00:47:29,895] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.11 | optimizer_gradients: 9.65 | optimizer_step: 9.90
[2026-02-24 00:47:29,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 546.50 | bwd_microstep: 2762.54 | bwd_inner_microstep: 2747.59 | bwd_allreduce_microstep: 14.91 | step_microstep: 90.08
[2026-02-24 00:47:29,896] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 1090.44 | bwd: 3452.82 | bwd_inner: 3437.84 | bwd_allreduce: 14.93 | step: 90.16

 27%|██▋       | 24/90 [02:39<04:36,  4.19s/it]
                                               
{'loss': 0.1028, 'learning_rate': 1.725995491923131e-05, 'epoch': 0.8}

 27%|██▋       | 24/90 [02:39<04:36,  4.19s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:47:30,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.93 | bwd_microstep: 503.71 | bwd_inner_microstep: 503.68 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.06
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:47:35,820] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.80 | optimizer_gradients: 9.65 | optimizer_step: 9.88
[2026-02-24 00:47:35,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 325.19 | bwd_microstep: 4628.92 | bwd_inner_microstep: 4584.77 | bwd_allreduce_microstep: 44.00 | step_microstep: 91.20
[2026-02-24 00:47:35,821] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 649.12 | bwd: 5132.64 | bwd_inner: 5088.55 | bwd_allreduce: 44.00 | step: 91.28

 28%|██▊       | 25/90 [02:45<05:06,  4.71s/it]
                                               
{'loss': 0.1697, 'learning_rate': 1.7006946020733426e-05, 'epoch': 0.83}

 28%|██▊       | 25/90 [02:45<05:06,  4.71s/it]dynamic ViT batch size: 244, images per sample: 244.0, dynamic token length: 2419
[2026-02-24 00:47:36,590] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 290.91 | bwd_microstep: 447.77 | bwd_inner_microstep: 447.75 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.09
dynamic ViT batch size: 248, images per sample: 248.0, dynamic token length: 2423
[2026-02-24 00:47:38,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 59.05 | optimizer_gradients: 9.65 | optimizer_step: 9.87
[2026-02-24 00:47:38,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 294.51 | bwd_microstep: 1413.79 | bwd_inner_microstep: 1369.51 | bwd_allreduce_microstep: 44.24 | step_microstep: 91.04
[2026-02-24 00:47:38,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 585.42 | bwd: 1861.57 | bwd_inner: 1817.26 | bwd_allreduce: 44.26 | step: 91.14

 29%|██▉       | 26/90 [02:47<04:20,  4.07s/it]
                                               
{'loss': 0.0879, 'learning_rate': 1.6744801401405138e-05, 'epoch': 0.87}

 29%|██▉       | 26/90 [02:47<04:20,  4.07s/it]dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2979
[2026-02-24 00:47:39,679] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 543.71 | bwd_microstep: 690.35 | bwd_inner_microstep: 690.32 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.06
dynamic ViT batch size: 512, images per sample: 512.0, dynamic token length: 2687
[2026-02-24 00:47:44,155] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.22 | optimizer_gradients: 9.61 | optimizer_step: 9.90
[2026-02-24 00:47:44,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 508.69 | bwd_microstep: 3849.01 | bwd_inner_microstep: 3826.18 | bwd_allreduce_microstep: 22.76 | step_microstep: 90.46
[2026-02-24 00:47:44,156] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 1052.40 | bwd: 4539.36 | bwd_inner: 4516.52 | bwd_allreduce: 22.77 | step: 90.54

 30%|███       | 27/90 [02:53<04:48,  4.58s/it]
                                               
{'loss': 0.0679, 'learning_rate': 1.647386284781828e-05, 'epoch': 0.9}

 30%|███       | 27/90 [02:53<04:48,  4.58s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:47:45,007] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.64 | bwd_microstep: 501.59 | bwd_inner_microstep: 501.57 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.03
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:47:48,096] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.99 | optimizer_gradients: 9.60 | optimizer_step: 9.89
[2026-02-24 00:47:48,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.02 | bwd_microstep: 875.50 | bwd_inner_microstep: 830.87 | bwd_allreduce_microstep: 44.55 | step_microstep: 89.11
[2026-02-24 00:47:48,097] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 645.66 | bwd: 1377.10 | bwd_inner: 1332.46 | bwd_allreduce: 44.41 | step: 89.15

 31%|███       | 28/90 [02:57<04:31,  4.39s/it]
                                               
{'loss': 0.1246, 'learning_rate': 1.6194483612157232e-05, 'epoch': 0.93}

 31%|███       | 28/90 [02:57<04:31,  4.39s/it]dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2979
[2026-02-24 00:47:49,367] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 542.90 | bwd_microstep: 690.38 | bwd_inner_microstep: 690.36 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.04
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:47:56,945] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.11 | optimizer_gradients: 9.52 | optimizer_step: 9.86
[2026-02-24 00:47:56,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.79 | bwd_microstep: 7148.87 | bwd_inner_microstep: 7106.54 | bwd_allreduce_microstep: 42.30 | step_microstep: 89.12
[2026-02-24 00:47:56,946] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 865.69 | bwd: 7839.25 | bwd_inner: 7796.89 | bwd_allreduce: 42.31 | step: 89.17

 32%|███▏      | 29/90 [03:06<05:49,  5.72s/it]
                                               
{'loss': 0.0701, 'learning_rate': 1.590702795164551e-05, 'epoch': 0.97}

 32%|███▏      | 29/90 [03:06<05:49,  5.72s/it]dynamic ViT batch size: 264, images per sample: 264.0, dynamic token length: 2439
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
[2026-02-24 00:47:57,736] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.91 | bwd_microstep: 462.69 | bwd_inner_microstep: 462.67 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.03
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
dynamic ViT batch size: 240, images per sample: 240.0, dynamic token length: 2415
[2026-02-24 00:48:01,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.58 | optimizer_gradients: 9.47 | optimizer_step: 9.91
[2026-02-24 00:48:01,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 287.18 | bwd_microstep: 1716.11 | bwd_inner_microstep: 1672.26 | bwd_allreduce_microstep: 43.82 | step_microstep: 89.41
[2026-02-24 00:48:01,324] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 592.09 | bwd: 2178.81 | bwd_inner: 2134.93 | bwd_allreduce: 43.83 | step: 89.45

 33%|███▎      | 30/90 [03:10<05:19,  5.32s/it]
                                               
{'loss': 0.1517, 'learning_rate': 1.5611870653623826e-05, 'epoch': 1.0}

 33%|███▎      | 30/90 [03:10<05:19,  5.32s/it][2026-02-24 00:48:03,727] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:03,730] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:03,731] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:03,738] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:03,749] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:03,751] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:03,751] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:03,752] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:08,867] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:08,882] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:08,895] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:08,898] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:08,909] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:08,913] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:08,914] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:08,914] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:14,034] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:14,038] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:14,049] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:14,059] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:14,068] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:14,072] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:14,072] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:14,073] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:18,574] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:19,102] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:19,123] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:19,136] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:19,136] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:19,141] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:19,142] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:48:19,142] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[h264 @ 0xda460c0] mmco: unref short failure
[h264 @ 0xda460c0] mmco: unref short failure
dynamic ViT batch size: 256, images per sample: 256.0, dynamic token length: 2431
[2026-02-24 00:48:32,116] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.59 | bwd_microstep: 465.72 | bwd_inner_microstep: 465.69 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.20
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:48:45,952] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.99 | optimizer_gradients: 9.57 | optimizer_step: 9.90
[2026-02-24 00:48:45,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 327.56 | bwd_microstep: 13393.61 | bwd_inner_microstep: 13349.05 | bwd_allreduce_microstep: 44.49 | step_microstep: 91.15
[2026-02-24 00:48:45,953] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 639.13 | bwd: 13859.34 | bwd_inner: 13814.77 | bwd_allreduce: 44.49 | step: 91.38

 34%|███▍      | 31/90 [03:55<16:49, 17.11s/it]
                                               
{'loss': 0.0822, 'learning_rate': 1.530939654689887e-05, 'epoch': 1.03}

 34%|███▍      | 31/90 [03:55<16:49, 17.11s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2443
[2026-02-24 00:48:46,756] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 310.69 | bwd_microstep: 464.71 | bwd_inner_microstep: 464.68 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.04
dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2979
[2026-02-24 00:48:48,811] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.46 | optimizer_gradients: 9.61 | optimizer_step: 9.90
[2026-02-24 00:48:48,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 544.23 | bwd_microstep: 1390.90 | bwd_inner_microstep: 1375.97 | bwd_allreduce_microstep: 14.89 | step_microstep: 90.46
[2026-02-24 00:48:48,812] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 854.92 | bwd: 1855.61 | bwd_inner: 1840.65 | bwd_allreduce: 14.91 | step: 90.51

 36%|███▌      | 32/90 [03:58<12:24, 12.84s/it]
                                               
{'loss': 0.1023, 'learning_rate': 1.5000000000000002e-05, 'epoch': 1.07}

 36%|███▌      | 32/90 [03:58<12:24, 12.84s/it]dynamic ViT batch size: 296, images per sample: 296.0, dynamic token length: 2735
[2026-02-24 00:48:49,717] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 346.44 | bwd_microstep: 529.91 | bwd_inner_microstep: 529.89 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.10
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:48:51,467] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.41 | optimizer_gradients: 9.64 | optimizer_step: 9.87
[2026-02-24 00:48:51,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.97 | bwd_microstep: 1316.57 | bwd_inner_microstep: 1275.22 | bwd_allreduce_microstep: 41.18 | step_microstep: 90.58
[2026-02-24 00:48:51,468] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 670.41 | bwd: 1846.48 | bwd_inner: 1805.13 | bwd_allreduce: 41.19 | step: 90.74

 37%|███▋      | 33/90 [04:00<09:17,  9.78s/it]
                                               
{'loss': 0.0614, 'learning_rate': 1.4684084406997903e-05, 'epoch': 1.1}

 37%|███▋      | 33/90 [04:00<09:17,  9.78s/it]dynamic ViT batch size: 248, images per sample: 248.0, dynamic token length: 2423
[2026-02-24 00:48:52,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.34 | bwd_microstep: 450.32 | bwd_inner_microstep: 450.29 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.04
dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2979
[2026-02-24 00:48:54,145] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.24 | optimizer_gradients: 9.66 | optimizer_step: 9.86
[2026-02-24 00:48:54,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 544.73 | bwd_microstep: 1237.92 | bwd_inner_microstep: 1222.87 | bwd_allreduce_microstep: 15.01 | step_microstep: 90.31
[2026-02-24 00:48:54,146] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 840.06 | bwd: 1688.24 | bwd_inner: 1673.16 | bwd_allreduce: 15.03 | step: 90.36

 38%|███▊      | 34/90 [04:03<07:08,  7.65s/it]
                                               
{'loss': 0.1775, 'learning_rate': 1.4362061661555675e-05, 'epoch': 1.13}

 38%|███▊      | 34/90 [04:03<07:08,  7.65s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:48:54,999] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.38 | bwd_microstep: 502.24 | bwd_inner_microstep: 502.21 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.09
dynamic ViT batch size: 272, images per sample: 272.0, dynamic token length: 2711
[2026-02-24 00:49:04,776] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.32 | optimizer_gradients: 9.62 | optimizer_step: 9.92
[2026-02-24 00:49:04,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 327.64 | bwd_microstep: 9339.93 | bwd_inner_microstep: 9295.63 | bwd_allreduce_microstep: 44.23 | step_microstep: 90.23
[2026-02-24 00:49:04,777] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 651.01 | bwd: 9842.17 | bwd_inner: 9797.86 | bwd_allreduce: 44.24 | step: 90.34

 39%|███▉      | 35/90 [04:14<07:49,  8.55s/it]
                                               
{'loss': 0.1772, 'learning_rate': 1.4034351619898088e-05, 'epoch': 1.17}

 39%|███▉      | 35/90 [04:14<07:49,  8.55s/it]dynamic ViT batch size: 248, images per sample: 248.0, dynamic token length: 2423
[2026-02-24 00:49:05,547] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.43 | bwd_microstep: 449.97 | bwd_inner_microstep: 449.95 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.21
dynamic ViT batch size: 544, images per sample: 544.0, dynamic token length: 2983
[2026-02-24 00:49:07,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 60.16 | optimizer_gradients: 9.74 | optimizer_step: 9.90
[2026-02-24 00:49:07,217] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 546.06 | bwd_microstep: 988.42 | bwd_inner_microstep: 973.39 | bwd_allreduce_microstep: 14.99 | step_microstep: 103.82
[2026-02-24 00:49:07,218] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 839.50 | bwd: 1438.40 | bwd_inner: 1423.34 | bwd_allreduce: 15.01 | step: 104.04

 40%|████      | 36/90 [04:16<06:02,  6.71s/it]
                                               
{'loss': 0.177, 'learning_rate': 1.3701381553399147e-05, 'epoch': 1.2}

 40%|████      | 36/90 [04:16<06:02,  6.71s/it]dynamic ViT batch size: 272, images per sample: 272.0, dynamic token length: 2447
[2026-02-24 00:49:08,035] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 314.94 | bwd_microstep: 471.21 | bwd_inner_microstep: 471.19 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.05
dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2979
[2026-02-24 00:49:10,761] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 59.21 | optimizer_gradients: 9.66 | optimizer_step: 9.88
[2026-02-24 00:49:10,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 543.86 | bwd_microstep: 2057.04 | bwd_inner_microstep: 2041.61 | bwd_allreduce_microstep: 15.38 | step_microstep: 94.15
[2026-02-24 00:49:10,762] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 858.80 | bwd: 2528.26 | bwd_inner: 2512.80 | bwd_allreduce: 15.40 | step: 94.21

 41%|████      | 37/90 [04:20<05:05,  5.76s/it]
                                               
{'loss': 0.0634, 'learning_rate': 1.3363585591501751e-05, 'epoch': 1.23}

 41%|████      | 37/90 [04:20<05:05,  5.76s/it]dynamic ViT batch size: 536, images per sample: 536.0, dynamic token length: 2711
[2026-02-24 00:49:14,682] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 529.62 | bwd_microstep: 660.45 | bwd_inner_microstep: 660.43 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.07
dynamic ViT batch size: 536, images per sample: 536.0, dynamic token length: 2711
[2026-02-24 00:49:16,880] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.39 | optimizer_gradients: 9.64 | optimizer_step: 9.88
[2026-02-24 00:49:16,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 530.01 | bwd_microstep: 1545.64 | bwd_inner_microstep: 1530.54 | bwd_allreduce_microstep: 15.07 | step_microstep: 90.67
[2026-02-24 00:49:16,881] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 1059.63 | bwd: 2206.10 | bwd_inner: 2190.96 | bwd_allreduce: 15.09 | step: 90.75

 42%|████▏     | 38/90 [04:26<05:05,  5.87s/it]
                                               
{'loss': 0.0735, 'learning_rate': 1.3021404155695728e-05, 'epoch': 1.27}

 42%|████▏     | 38/90 [04:26<05:05,  5.87s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:49:17,735] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.58 | bwd_microstep: 504.13 | bwd_inner_microstep: 504.11 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.04
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[h264 @ 0xd485480] mmco: unref short failure
[h264 @ 0xd485480] mmco: unref short failure
[2026-02-24 00:49:25,339] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.25 | optimizer_gradients: 9.54 | optimizer_step: 9.90
[2026-02-24 00:49:25,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.42 | bwd_microstep: 7172.18 | bwd_inner_microstep: 7127.62 | bwd_allreduce_microstep: 44.48 | step_microstep: 90.18
[2026-02-24 00:49:25,340] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 645.00 | bwd: 7676.32 | bwd_inner: 7631.76 | bwd_allreduce: 44.40 | step: 90.23

 43%|████▎     | 39/90 [04:34<05:38,  6.65s/it]
                                               
{'loss': 0.2357, 'learning_rate': 1.2675283385292212e-05, 'epoch': 1.3}

 43%|████▎     | 39/90 [04:34<05:38,  6.65s/it]dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2979
[2026-02-24 00:49:27,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 542.67 | bwd_microstep: 690.14 | bwd_inner_microstep: 690.12 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.04
dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2979
[2026-02-24 00:49:28,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.19 | optimizer_gradients: 9.63 | optimizer_step: 9.91
[2026-02-24 00:49:28,926] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 543.31 | bwd_microstep: 776.99 | bwd_inner_microstep: 761.91 | bwd_allreduce_microstep: 15.04 | step_microstep: 89.86
[2026-02-24 00:49:28,927] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 1085.97 | bwd: 1467.14 | bwd_inner: 1452.03 | bwd_allreduce: 15.06 | step: 89.91

 44%|████▍     | 40/90 [04:38<04:46,  5.73s/it]
                                               
{'loss': 0.2736, 'learning_rate': 1.2325674555743106e-05, 'epoch': 1.33}

 44%|████▍     | 40/90 [04:38<04:46,  5.73s/it]dynamic ViT batch size: 240, images per sample: 240.0, dynamic token length: 2415
[2026-02-24 00:49:29,691] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 287.62 | bwd_microstep: 448.70 | bwd_inner_microstep: 448.68 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.06
dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2715
[2026-02-24 00:49:31,664] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.35 | optimizer_gradients: 9.62 | optimizer_step: 9.88
[2026-02-24 00:49:31,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 534.61 | bwd_microstep: 1317.73 | bwd_inner_microstep: 1290.65 | bwd_allreduce_microstep: 27.04 | step_microstep: 90.66
[2026-02-24 00:49:31,665] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 822.23 | bwd: 1766.44 | bwd_inner: 1739.33 | bwd_allreduce: 27.06 | step: 90.78

 46%|████▌     | 41/90 [04:41<03:56,  4.83s/it]
                                               
{'loss': 0.2299, 'learning_rate': 1.1973033490264e-05, 'epoch': 1.37}

 46%|████▌     | 41/90 [04:41<03:56,  4.83s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:49:32,519] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.57 | bwd_microstep: 503.67 | bwd_inner_microstep: 503.64 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.04
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:49:39,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.12 | optimizer_gradients: 9.61 | optimizer_step: 9.89
[2026-02-24 00:49:39,370] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.73 | bwd_microstep: 6416.31 | bwd_inner_microstep: 6371.88 | bwd_allreduce_microstep: 44.38 | step_microstep: 89.75
[2026-02-24 00:49:39,371] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 648.29 | bwd: 6919.99 | bwd_inner: 6875.53 | bwd_allreduce: 44.39 | step: 89.80

 47%|████▋     | 42/90 [04:48<04:33,  5.69s/it]
                                               
{'loss': 0.1089, 'learning_rate': 1.161781996552765e-05, 'epoch': 1.4}

 47%|████▋     | 42/90 [04:48<04:33,  5.69s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:49:40,231] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 325.21 | bwd_microstep: 501.90 | bwd_inner_microstep: 501.88 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.09
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:49:41,883] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.54 | optimizer_gradients: 9.81 | optimizer_step: 9.88
[2026-02-24 00:49:41,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.69 | bwd_microstep: 1215.60 | bwd_inner_microstep: 1172.08 | bwd_allreduce_microstep: 43.46 | step_microstep: 92.83
[2026-02-24 00:49:41,884] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 648.90 | bwd: 1717.51 | bwd_inner: 1673.97 | bwd_allreduce: 43.37 | step: 92.94

 48%|████▊     | 43/90 [04:51<03:42,  4.74s/it]
                                               
{'loss': 0.1255, 'learning_rate': 1.1260497112202895e-05, 'epoch': 1.43}

 48%|████▊     | 43/90 [04:51<03:42,  4.74s/it]dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2715
[2026-02-24 00:49:43,124] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 536.07 | bwd_microstep: 664.82 | bwd_inner_microstep: 664.79 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.07
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:49:44,195] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.58 | optimizer_gradients: 9.78 | optimizer_step: 9.90
[2026-02-24 00:49:44,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 325.60 | bwd_microstep: 633.70 | bwd_inner_microstep: 613.65 | bwd_allreduce_microstep: 20.00 | step_microstep: 91.83
[2026-02-24 00:49:44,196] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 861.67 | bwd: 1298.52 | bwd_inner: 1278.45 | bwd_allreduce: 20.02 | step: 91.97

 49%|████▉     | 44/90 [04:53<03:04,  4.01s/it]
                                               
{'loss': 0.0854, 'learning_rate': 1.0901530811120655e-05, 'epoch': 1.47}

 49%|████▉     | 44/90 [04:53<03:04,  4.01s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:49:45,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.18 | bwd_microstep: 503.84 | bwd_inner_microstep: 503.81 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.04
dynamic ViT batch size: 256, images per sample: 256.0, dynamic token length: 2431
[2026-02-24 00:49:50,918] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.32 | optimizer_gradients: 9.65 | optimizer_step: 9.87
[2026-02-24 00:49:50,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 302.04 | bwd_microstep: 5451.61 | bwd_inner_microstep: 5409.83 | bwd_allreduce_microstep: 41.73 | step_microstep: 90.67
[2026-02-24 00:49:50,919] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 626.22 | bwd: 5955.46 | bwd_inner: 5913.64 | bwd_allreduce: 41.74 | step: 90.73

 50%|█████     | 45/90 [05:00<03:37,  4.82s/it]
                                               
{'loss': 0.1109, 'learning_rate': 1.0541389085854177e-05, 'epoch': 1.5}

 50%|█████     | 45/90 [05:00<03:37,  4.82s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:49:51,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 325.70 | bwd_microstep: 505.91 | bwd_inner_microstep: 505.88 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.09
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:49:58,930] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.57 | optimizer_gradients: 9.64 | optimizer_step: 9.87
[2026-02-24 00:49:58,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.25 | bwd_microstep: 6710.91 | bwd_inner_microstep: 6667.81 | bwd_allreduce_microstep: 43.02 | step_microstep: 93.47
[2026-02-24 00:49:58,932] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 649.95 | bwd: 7216.83 | bwd_inner: 7173.72 | bwd_allreduce: 43.03 | step: 93.57

 51%|█████     | 46/90 [05:08<04:14,  5.78s/it]
                                               
{'loss': 0.0703, 'learning_rate': 1.0180541492505605e-05, 'epoch': 1.53}

 51%|█████     | 46/90 [05:08<04:14,  5.78s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:49:59,785] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.47 | bwd_microstep: 502.89 | bwd_inner_microstep: 502.86 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.06
dynamic ViT batch size: 256, images per sample: 256.0, dynamic token length: 2431
[2026-02-24 00:50:01,241] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.30 | optimizer_gradients: 9.70 | optimizer_step: 9.89
[2026-02-24 00:50:01,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.41 | bwd_microstep: 1045.66 | bwd_inner_microstep: 1002.31 | bwd_allreduce_microstep: 43.28 | step_microstep: 91.15
[2026-02-24 00:50:01,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 625.88 | bwd: 1548.55 | bwd_inner: 1505.19 | bwd_allreduce: 43.16 | step: 91.22

 52%|█████▏    | 47/90 [05:10<03:23,  4.74s/it]
                                               
{'loss': 0.0915, 'learning_rate': 9.819458507494395e-06, 'epoch': 1.57}

 52%|█████▏    | 47/90 [05:10<03:23,  4.74s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2443
[2026-02-24 00:50:02,053] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 312.80 | bwd_microstep: 467.76 | bwd_inner_microstep: 467.73 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.15
dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2979
[2026-02-24 00:50:03,781] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.22 | optimizer_gradients: 9.73 | optimizer_step: 9.89
[2026-02-24 00:50:03,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 554.86 | bwd_microstep: 1052.26 | bwd_inner_microstep: 1037.05 | bwd_allreduce_microstep: 15.17 | step_microstep: 90.42
[2026-02-24 00:50:03,782] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 867.66 | bwd: 1520.03 | bwd_inner: 1504.78 | bwd_allreduce: 15.19 | step: 90.60

 53%|█████▎    | 48/90 [05:13<02:51,  4.08s/it]
                                               
{'loss': 0.1186, 'learning_rate': 9.458610914145826e-06, 'epoch': 1.6}

 53%|█████▎    | 48/90 [05:13<02:51,  4.08s/it]dynamic ViT batch size: 244, images per sample: 244.0, dynamic token length: 2419
[2026-02-24 00:50:04,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.35 | bwd_microstep: 450.49 | bwd_inner_microstep: 450.47 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.09
dynamic ViT batch size: 252, images per sample: 252.0, dynamic token length: 2427
[2026-02-24 00:50:06,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.55 | optimizer_gradients: 9.69 | optimizer_step: 9.88
[2026-02-24 00:50:06,554] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 300.32 | bwd_microstep: 1589.41 | bwd_inner_microstep: 1552.18 | bwd_allreduce_microstep: 37.16 | step_microstep: 91.05
[2026-02-24 00:50:06,555] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 591.68 | bwd: 2039.91 | bwd_inner: 2002.67 | bwd_allreduce: 37.17 | step: 91.15

 54%|█████▍    | 49/90 [05:16<02:31,  3.69s/it]
                                               
{'loss': 0.1311, 'learning_rate': 9.098469188879348e-06, 'epoch': 1.63}

 54%|█████▍    | 49/90 [05:16<02:31,  3.69s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:50:07,418] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.98 | bwd_microstep: 508.32 | bwd_inner_microstep: 508.29 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.05
dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2979
[2026-02-24 00:50:15,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.12 | optimizer_gradients: 9.67 | optimizer_step: 9.91
[2026-02-24 00:50:15,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 547.76 | bwd_microstep: 7162.86 | bwd_inner_microstep: 7145.88 | bwd_allreduce_microstep: 16.95 | step_microstep: 90.59
[2026-02-24 00:50:15,906] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 872.74 | bwd: 7671.19 | bwd_inner: 7654.17 | bwd_allreduce: 16.97 | step: 90.65

 56%|█████▌    | 50/90 [05:25<03:35,  5.39s/it]
                                               
{'loss': 0.0585, 'learning_rate': 8.739502887797108e-06, 'epoch': 1.67}

 56%|█████▌    | 50/90 [05:25<03:35,  5.39s/it]dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2979
[2026-02-24 00:50:17,224] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 549.18 | bwd_microstep: 727.56 | bwd_inner_microstep: 727.53 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.33
dynamic ViT batch size: 272, images per sample: 272.0, dynamic token length: 2711
[2026-02-24 00:50:19,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.44 | optimizer_gradients: 9.69 | optimizer_step: 9.87
[2026-02-24 00:50:19,425] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 384.47 | bwd_microstep: 1701.28 | bwd_inner_microstep: 1657.72 | bwd_allreduce_microstep: 43.52 | step_microstep: 90.62
[2026-02-24 00:50:19,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 933.66 | bwd: 2428.84 | bwd_inner: 2385.25 | bwd_allreduce: 43.43 | step: 90.96

 57%|█████▋    | 51/90 [05:28<03:08,  4.83s/it]
                                               
{'loss': 0.1137, 'learning_rate': 8.382180034472353e-06, 'epoch': 1.7}

 57%|█████▋    | 51/90 [05:28<03:08,  4.83s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:50:20,280] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 325.00 | bwd_microstep: 502.71 | bwd_inner_microstep: 502.69 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.05
dynamic ViT batch size: 536, images per sample: 536.0, dynamic token length: 2711
[2026-02-24 00:50:26,051] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.05 | optimizer_gradients: 10.04 | optimizer_step: 9.90
[2026-02-24 00:50:26,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 531.30 | bwd_microstep: 5115.48 | bwd_inner_microstep: 5072.98 | bwd_allreduce_microstep: 42.45 | step_microstep: 93.47
[2026-02-24 00:50:26,052] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 856.30 | bwd: 5618.20 | bwd_inner: 5575.67 | bwd_allreduce: 42.48 | step: 93.53

 58%|█████▊    | 52/90 [05:35<03:23,  5.37s/it]
                                               
{'loss': 0.0864, 'learning_rate': 8.026966509736001e-06, 'epoch': 1.73}

 58%|█████▊    | 52/90 [05:35<03:23,  5.37s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:50:26,905] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.27 | bwd_microstep: 503.83 | bwd_inner_microstep: 503.81 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.05
dynamic ViT batch size: 288, images per sample: 288.0, dynamic token length: 2727
[2026-02-24 00:50:33,632] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.36 | optimizer_gradients: 9.61 | optimizer_step: 9.86
[2026-02-24 00:50:33,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 340.33 | bwd_microstep: 6276.31 | bwd_inner_microstep: 6232.91 | bwd_allreduce_microstep: 43.36 | step_microstep: 90.13
[2026-02-24 00:50:33,633] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 663.59 | bwd: 6780.15 | bwd_inner: 6736.72 | bwd_allreduce: 43.37 | step: 90.18

 59%|█████▉    | 53/90 [05:43<03:43,  6.03s/it]
                                               
{'loss': 0.1005, 'learning_rate': 7.674325444256899e-06, 'epoch': 1.77}

 59%|█████▉    | 53/90 [05:43<03:43,  6.03s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:50:34,484] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.11 | bwd_microstep: 501.13 | bwd_inner_microstep: 501.10 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.04
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:50:36,600] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.12 | optimizer_gradients: 9.72 | optimizer_step: 9.90
[2026-02-24 00:50:36,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.41 | bwd_microstep: 1683.60 | bwd_inner_microstep: 1639.77 | bwd_allreduce_microstep: 43.75 | step_microstep: 90.55
[2026-02-24 00:50:36,601] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 646.52 | bwd: 2184.73 | bwd_inner: 2140.90 | bwd_allreduce: 43.76 | step: 90.60

 60%|██████    | 54/90 [05:46<03:04,  5.11s/it]
                                               
{'loss': 0.072, 'learning_rate': 7.324716614707794e-06, 'epoch': 1.8}

 60%|██████    | 54/90 [05:46<03:04,  5.11s/it]dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2715
[2026-02-24 00:50:37,835] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 533.85 | bwd_microstep: 663.80 | bwd_inner_microstep: 663.78 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.07
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:50:39,077] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.94 | optimizer_gradients: 9.80 | optimizer_step: 9.89
[2026-02-24 00:50:39,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.40 | bwd_microstep: 808.22 | bwd_inner_microstep: 764.98 | bwd_allreduce_microstep: 43.16 | step_microstep: 90.16
[2026-02-24 00:50:39,078] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 858.25 | bwd: 1472.03 | bwd_inner: 1428.79 | bwd_allreduce: 43.17 | step: 90.24

 61%|██████    | 55/90 [05:48<02:31,  4.32s/it]
                                               
{'loss': 0.0638, 'learning_rate': 6.978595844304272e-06, 'epoch': 1.83}

 61%|██████    | 55/90 [05:48<02:31,  4.32s/it]dynamic ViT batch size: 516, images per sample: 516.0, dynamic token length: 2691
[2026-02-24 00:50:40,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 515.26 | bwd_microstep: 653.61 | bwd_inner_microstep: 653.58 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.36
dynamic ViT batch size: 248, images per sample: 248.0, dynamic token length: 2423
[2026-02-24 00:50:41,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.31 | optimizer_gradients: 9.67 | optimizer_step: 9.90
[2026-02-24 00:50:41,575] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.60 | bwd_microstep: 882.51 | bwd_inner_microstep: 841.02 | bwd_allreduce_microstep: 41.41 | step_microstep: 90.56
[2026-02-24 00:50:41,576] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 808.86 | bwd: 1536.12 | bwd_inner: 1494.63 | bwd_allreduce: 41.42 | step: 90.94

 62%|██████▏   | 56/90 [05:51<02:08,  3.77s/it]
                                               
{'loss': 0.0857, 'learning_rate': 6.636414408498249e-06, 'epoch': 1.87}

 62%|██████▏   | 56/90 [05:51<02:08,  3.77s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:50:42,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.16 | bwd_microstep: 502.34 | bwd_inner_microstep: 502.32 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.08
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:50:48,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.26 | optimizer_gradients: 9.56 | optimizer_step: 9.85
[2026-02-24 00:50:48,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.35 | bwd_microstep: 5225.95 | bwd_inner_microstep: 5210.42 | bwd_allreduce_microstep: 15.47 | step_microstep: 90.04
[2026-02-24 00:50:48,228] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 644.51 | bwd: 5728.30 | bwd_inner: 5712.76 | bwd_allreduce: 15.47 | step: 90.12

 63%|██████▎   | 57/90 [05:57<02:33,  4.64s/it]
                                               
{'loss': 0.1047, 'learning_rate': 6.298618446600856e-06, 'epoch': 1.9}

 63%|██████▎   | 57/90 [05:57<02:33,  4.64s/it]dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2979
[2026-02-24 00:50:49,498] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 542.52 | bwd_microstep: 690.39 | bwd_inner_microstep: 690.37 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.03
dynamic ViT batch size: 244, images per sample: 244.0, dynamic token length: 2419
[2026-02-24 00:50:50,509] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.55 | optimizer_gradients: 9.52 | optimizer_step: 9.89
[2026-02-24 00:50:50,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 289.59 | bwd_microstep: 615.42 | bwd_inner_microstep: 570.18 | bwd_allreduce_microstep: 45.21 | step_microstep: 89.53
[2026-02-24 00:50:50,510] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 832.10 | bwd: 1305.82 | bwd_inner: 1260.54 | bwd_allreduce: 45.23 | step: 89.57

 64%|██████▍   | 58/90 [05:59<02:05,  3.93s/it]
                                               
{'loss': 0.0901, 'learning_rate': 5.965648380101916e-06, 'epoch': 1.93}

 64%|██████▍   | 58/90 [05:59<02:05,  3.93s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:50:51,357] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.82 | bwd_microstep: 501.76 | bwd_inner_microstep: 501.73 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.03
dynamic ViT batch size: 500, images per sample: 500.0, dynamic token length: 2675
[2026-02-24 00:50:56,551] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.24 | optimizer_gradients: 9.56 | optimizer_step: 9.88
[2026-02-24 00:50:56,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 499.12 | bwd_microstep: 4577.58 | bwd_inner_microstep: 4532.99 | bwd_allreduce_microstep: 44.53 | step_microstep: 89.22
[2026-02-24 00:50:56,552] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 821.93 | bwd: 5079.34 | bwd_inner: 5034.74 | bwd_allreduce: 44.53 | step: 89.25

 66%|██████▌   | 59/90 [06:06<02:21,  4.56s/it]
                                               
{'loss': 0.0626, 'learning_rate': 5.637938338444325e-06, 'epoch': 1.97}

 66%|██████▌   | 59/90 [06:06<02:21,  4.56s/it]dynamic ViT batch size: 248, images per sample: 248.0, dynamic token length: 2423
[2026-02-24 00:50:57,313] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 291.39 | bwd_microstep: 447.87 | bwd_inner_microstep: 447.84 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.03
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
dynamic ViT batch size: 536, images per sample: 536.0, dynamic token length: 2711
[2026-02-24 00:51:01,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.41 | optimizer_gradients: 9.46 | optimizer_step: 9.90
[2026-02-24 00:51:01,118] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 530.33 | bwd_microstep: 1792.67 | bwd_inner_microstep: 1758.59 | bwd_allreduce_microstep: 34.05 | step_microstep: 89.25
[2026-02-24 00:51:01,119] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 821.72 | bwd: 2240.55 | bwd_inner: 2206.43 | bwd_allreduce: 34.07 | step: 89.28

 67%|██████▋   | 60/90 [06:10<02:16,  4.56s/it]
                                               
{'loss': 0.0482, 'learning_rate': 5.3159155930021e-06, 'epoch': 2.0}

 67%|██████▋   | 60/90 [06:10<02:16,  4.56s/it][2026-02-24 00:51:03,470] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:03,523] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:03,523] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:03,910] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:03,917] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:03,929] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:03,931] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:03,931] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:10,249] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:10,257] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:10,258] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:10,264] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:10,563] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:10,565] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:10,568] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:10,569] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:14,503] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:14,743] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:14,754] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:14,761] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:15,355] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:15,635] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:15,637] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:15,638] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:19,174] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:19,177] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:19,180] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:19,255] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:19,695] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:19,863] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:20,569] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-24 00:51:20,680] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
dynamic ViT batch size: 248, images per sample: 248.0, dynamic token length: 2423
[2026-02-24 00:51:33,800] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 297.49 | bwd_microstep: 449.37 | bwd_inner_microstep: 449.35 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.04
dynamic ViT batch size: 272, images per sample: 272.0, dynamic token length: 2711
[h264 @ 0xd2ed5c0] mmco: unref short failure
[h264 @ 0xd2ed5c0] mmco: unref short failure
[2026-02-24 00:51:47,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.94 | optimizer_gradients: 9.52 | optimizer_step: 9.88
[2026-02-24 00:51:47,994] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 326.83 | bwd_microstep: 13755.41 | bwd_inner_microstep: 13711.37 | bwd_allreduce_microstep: 43.90 | step_microstep: 91.83
[2026-02-24 00:51:47,995] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 624.31 | bwd: 14204.79 | bwd_inner: 14160.81 | bwd_allreduce: 43.90 | step: 91.88

 68%|██████▊   | 61/90 [06:57<08:20, 17.26s/it]
                                               
{'loss': 0.092, 'learning_rate': 5.000000000000003e-06, 'epoch': 2.03}

 68%|██████▊   | 61/90 [06:57<08:20, 17.26s/it]dynamic ViT batch size: 276, images per sample: 276.0, dynamic token length: 2715
[2026-02-24 00:51:48,857] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 329.34 | bwd_microstep: 505.18 | bwd_inner_microstep: 505.16 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.04
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:51:50,788] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.55 | optimizer_gradients: 9.65 | optimizer_step: 9.86
[2026-02-24 00:51:50,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.44 | bwd_microstep: 1498.86 | bwd_inner_microstep: 1454.79 | bwd_allreduce_microstep: 43.89 | step_microstep: 91.01
[2026-02-24 00:51:50,789] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 652.76 | bwd: 2004.05 | bwd_inner: 1959.98 | bwd_allreduce: 43.87 | step: 91.06

 69%|██████▉   | 62/90 [07:00<06:01, 12.92s/it]
                                               
{'loss': 0.0455, 'learning_rate': 4.690603453101134e-06, 'epoch': 2.07}

 69%|██████▉   | 62/90 [07:00<06:01, 12.92s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:51:51,651] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 325.18 | bwd_microstep: 508.16 | bwd_inner_microstep: 508.13 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.11
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2443
[2026-02-24 00:52:00,054] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 395.43 | optimizer_gradients: 9.56 | optimizer_step: 9.86
[2026-02-24 00:52:00,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 311.49 | bwd_microstep: 7645.10 | bwd_inner_microstep: 7603.65 | bwd_allreduce_microstep: 41.38 | step_microstep: 427.97
[2026-02-24 00:52:00,055] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 636.64 | bwd: 8153.27 | bwd_inner: 8111.81 | bwd_allreduce: 41.39 | step: 428.09

 70%|███████   | 63/90 [07:09<05:19, 11.82s/it]
                                               
{'loss': 0.0437, 'learning_rate': 4.388129346376177e-06, 'epoch': 2.1}

 70%|███████   | 63/90 [07:09<05:19, 11.82s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:52:00,907] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.00 | bwd_microstep: 500.99 | bwd_inner_microstep: 500.96 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.07
dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2979
[2026-02-24 00:52:02,500] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.57 | optimizer_gradients: 9.67 | optimizer_step: 9.88
[2026-02-24 00:52:02,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 544.58 | bwd_microstep: 925.61 | bwd_inner_microstep: 909.42 | bwd_allreduce_microstep: 16.15 | step_microstep: 91.45
[2026-02-24 00:52:02,501] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 867.58 | bwd: 1426.60 | bwd_inner: 1410.38 | bwd_allreduce: 16.17 | step: 91.54

 71%|███████   | 64/90 [07:11<03:54,  9.01s/it]
                                               
{'loss': 0.0648, 'learning_rate': 4.092972048354491e-06, 'epoch': 2.13}

 71%|███████   | 64/90 [07:11<03:54,  9.01s/it]dynamic ViT batch size: 292, images per sample: 292.0, dynamic token length: 2731
[2026-02-24 00:52:03,406] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 343.12 | bwd_microstep: 536.57 | bwd_inner_microstep: 536.54 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.08
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:52:05,012] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.59 | optimizer_gradients: 10.07 | optimizer_step: 9.92
[2026-02-24 00:52:05,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.93 | bwd_microstep: 1165.57 | bwd_inner_microstep: 1121.27 | bwd_allreduce_microstep: 44.19 | step_microstep: 98.63
[2026-02-24 00:52:05,014] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 667.06 | bwd: 1702.15 | bwd_inner: 1657.84 | bwd_allreduce: 44.21 | step: 98.73

 72%|███████▏  | 65/90 [07:14<02:56,  7.06s/it]
                                               
{'loss': 0.0601, 'learning_rate': 3.8055163878427703e-06, 'epoch': 2.17}

 72%|███████▏  | 65/90 [07:14<02:56,  7.06s/it]dynamic ViT batch size: 252, images per sample: 252.0, dynamic token length: 2427
[2026-02-24 00:52:05,808] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 304.28 | bwd_microstep: 453.03 | bwd_inner_microstep: 453.01 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.05
dynamic ViT batch size: 264, images per sample: 264.0, dynamic token length: 2439
[h264 @ 0x957d580] mmco: unref short failure
[h264 @ 0x957d580] mmco: unref short failure
[2026-02-24 00:52:07,520] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.95 | optimizer_gradients: 9.65 | optimizer_step: 9.89
[2026-02-24 00:52:07,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 308.11 | bwd_microstep: 1292.53 | bwd_inner_microstep: 1249.30 | bwd_allreduce_microstep: 42.97 | step_microstep: 91.55
[2026-02-24 00:52:07,521] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 612.36 | bwd: 1745.57 | bwd_inner: 1702.47 | bwd_allreduce: 42.98 | step: 91.60

 73%|███████▎  | 66/90 [07:17<02:16,  5.69s/it]
                                               
{'loss': 0.0596, 'learning_rate': 3.5261371521817247e-06, 'epoch': 2.2}

 73%|███████▎  | 66/90 [07:17<02:16,  5.69s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:52:08,379] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.67 | bwd_microstep: 504.12 | bwd_inner_microstep: 504.09 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.05
dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2979
[2026-02-24 00:52:10,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 59.62 | optimizer_gradients: 9.70 | optimizer_step: 9.90
[2026-02-24 00:52:10,242] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 544.18 | bwd_microstep: 1192.48 | bwd_inner_microstep: 1168.84 | bwd_allreduce_microstep: 23.59 | step_microstep: 94.26
[2026-02-24 00:52:10,243] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 868.84 | bwd: 1696.60 | bwd_inner: 1672.93 | bwd_allreduce: 23.61 | step: 94.32

 74%|███████▍  | 67/90 [07:19<01:50,  4.80s/it]
                                               
{'loss': 0.0842, 'learning_rate': 3.255198598594862e-06, 'epoch': 2.23}

 74%|███████▍  | 67/90 [07:19<01:50,  4.80s/it]dynamic ViT batch size: 252, images per sample: 252.0, dynamic token length: 2427
[2026-02-24 00:52:11,025] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 298.13 | bwd_microstep: 455.53 | bwd_inner_microstep: 455.51 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.06
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:52:21,294] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 59.32 | optimizer_gradients: 9.67 | optimizer_step: 9.91
[2026-02-24 00:52:21,295] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 561.78 | bwd_microstep: 9593.35 | bwd_inner_microstep: 9571.56 | bwd_allreduce_microstep: 21.57 | step_microstep: 94.73
[2026-02-24 00:52:21,296] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 859.91 | bwd: 10048.89 | bwd_inner: 10027.10 | bwd_allreduce: 21.58 | step: 94.80

 76%|███████▌  | 68/90 [07:30<02:26,  6.68s/it]
                                               
{'loss': 0.0409, 'learning_rate': 2.9930539792665767e-06, 'epoch': 2.27}

 76%|███████▌  | 68/90 [07:30<02:26,  6.68s/it]dynamic ViT batch size: 288, images per sample: 288.0, dynamic token length: 2727
[2026-02-24 00:52:22,179] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 338.77 | bwd_microstep: 515.75 | bwd_inner_microstep: 515.73 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.04
dynamic ViT batch size: 252, images per sample: 252.0, dynamic token length: 2427
[2026-02-24 00:52:23,983] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.57 | optimizer_gradients: 9.60 | optimizer_step: 9.88
[2026-02-24 00:52:23,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 296.78 | bwd_microstep: 1398.16 | bwd_inner_microstep: 1354.28 | bwd_allreduce_microstep: 43.83 | step_microstep: 91.49
[2026-02-24 00:52:23,985] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 635.56 | bwd: 1913.92 | bwd_inner: 1870.01 | bwd_allreduce: 43.74 | step: 91.55

 77%|███████▋  | 69/90 [07:33<01:55,  5.48s/it]
                                               
{'loss': 0.0317, 'learning_rate': 2.740045080768694e-06, 'epoch': 2.3}

 77%|███████▋  | 69/90 [07:33<01:55,  5.48s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:52:24,844] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.03 | bwd_microstep: 504.00 | bwd_inner_microstep: 503.97 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.04
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:52:34,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 59.33 | optimizer_gradients: 9.54 | optimizer_step: 9.89
[2026-02-24 00:52:34,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.04 | bwd_microstep: 9304.34 | bwd_inner_microstep: 9261.13 | bwd_allreduce_microstep: 43.14 | step_microstep: 92.83
[2026-02-24 00:52:34,584] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 648.07 | bwd: 9808.34 | bwd_inner: 9765.12 | bwd_allreduce: 43.14 | step: 92.88

 78%|███████▊  | 70/90 [07:44<02:20,  7.02s/it]
                                               
{'loss': 0.0369, 'learning_rate': 2.496501778435977e-06, 'epoch': 2.33}

 78%|███████▊  | 70/90 [07:44<02:20,  7.02s/it]dynamic ViT batch size: 272, images per sample: 272.0, dynamic token length: 2711
[2026-02-24 00:52:35,444] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 326.46 | bwd_microstep: 505.80 | bwd_inner_microstep: 505.77 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.05
dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2715
[2026-02-24 00:52:38,167] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 59.20 | optimizer_gradients: 9.62 | optimizer_step: 9.90
[2026-02-24 00:52:38,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 534.65 | bwd_microstep: 2063.55 | bwd_inner_microstep: 2048.42 | bwd_allreduce_microstep: 15.10 | step_microstep: 93.40
[2026-02-24 00:52:38,168] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 861.10 | bwd: 2569.36 | bwd_inner: 2554.19 | bwd_allreduce: 15.12 | step: 93.52

 79%|███████▉  | 71/90 [07:47<01:53,  5.99s/it]
                                               
{'loss': 0.0569, 'learning_rate': 2.2627416062716366e-06, 'epoch': 2.37}

 79%|███████▉  | 71/90 [07:47<01:53,  5.99s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:52:39,040] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 337.99 | bwd_microstep: 507.27 | bwd_inner_microstep: 507.24 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.04
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:52:40,677] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.35 | optimizer_gradients: 10.28 | optimizer_step: 9.89
[2026-02-24 00:52:40,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.26 | bwd_microstep: 1188.74 | bwd_inner_microstep: 1154.51 | bwd_allreduce_microstep: 34.04 | step_microstep: 105.86
[2026-02-24 00:52:40,678] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 661.26 | bwd: 1696.04 | bwd_inner: 1661.81 | bwd_allreduce: 33.96 | step: 105.87

 80%|████████  | 72/90 [07:50<01:29,  4.95s/it]
                                               
{'loss': 0.0374, 'learning_rate': 2.0390693429435626e-06, 'epoch': 2.4}

 80%|████████  | 72/90 [07:50<01:29,  4.95s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:52:41,561] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 339.36 | bwd_microstep: 503.47 | bwd_inner_microstep: 503.44 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.15
dynamic ViT batch size: 252, images per sample: 252.0, dynamic token length: 2427
[2026-02-24 00:52:43,426] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.46 | optimizer_gradients: 10.02 | optimizer_step: 9.87
[2026-02-24 00:52:43,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 301.28 | bwd_microstep: 1448.92 | bwd_inner_microstep: 1405.67 | bwd_allreduce_microstep: 43.10 | step_microstep: 91.82
[2026-02-24 00:52:43,427] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 640.65 | bwd: 1952.40 | bwd_inner: 1909.12 | bwd_allreduce: 43.07 | step: 92.00

 81%|████████  | 73/90 [07:52<01:12,  4.28s/it]
                                               
{'loss': 0.0672, 'learning_rate': 1.8257766144110823e-06, 'epoch': 2.43}

 81%|████████  | 73/90 [07:52<01:12,  4.28s/it]dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2979
[2026-02-24 00:52:44,722] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 557.92 | bwd_microstep: 700.85 | bwd_inner_microstep: 700.82 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.08
dynamic ViT batch size: 536, images per sample: 536.0, dynamic token length: 2711
[2026-02-24 00:52:51,502] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.16 | optimizer_gradients: 9.69 | optimizer_step: 9.88
[2026-02-24 00:52:51,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 539.71 | bwd_microstep: 6118.49 | bwd_inner_microstep: 6085.40 | bwd_allreduce_microstep: 33.05 | step_microstep: 90.75
[2026-02-24 00:52:51,503] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 1097.62 | bwd: 6819.34 | bwd_inner: 6786.22 | bwd_allreduce: 33.07 | step: 90.85

 82%|████████▏ | 74/90 [08:00<01:26,  5.42s/it]
                                               
{'loss': 0.0777, 'learning_rate': 1.6231415137003536e-06, 'epoch': 2.47}

 82%|████████▏ | 74/90 [08:00<01:26,  5.42s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:52:52,358] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.01 | bwd_microstep: 501.38 | bwd_inner_microstep: 501.36 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.05
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:52:54,188] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.12 | optimizer_gradients: 9.69 | optimizer_step: 9.91
[2026-02-24 00:52:54,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.55 | bwd_microstep: 1397.72 | bwd_inner_microstep: 1353.67 | bwd_allreduce_microstep: 43.82 | step_microstep: 90.58
[2026-02-24 00:52:54,189] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 646.56 | bwd: 1899.11 | bwd_inner: 1855.05 | bwd_allreduce: 43.83 | step: 90.65

 83%|████████▎ | 75/90 [08:03<01:09,  4.60s/it]
                                               
{'loss': 0.0451, 'learning_rate': 1.4314282383241097e-06, 'epoch': 2.5}

 83%|████████▎ | 75/90 [08:03<01:09,  4.60s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:52:55,042] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.63 | bwd_microstep: 502.98 | bwd_inner_microstep: 502.96 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.10
dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2979
[2026-02-24 00:52:58,540] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 122.53 | optimizer_gradients: 21.45 | optimizer_step: 22.34
[2026-02-24 00:52:58,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 546.26 | bwd_microstep: 2729.23 | bwd_inner_microstep: 2710.79 | bwd_allreduce_microstep: 18.43 | step_microstep: 191.47
[2026-02-24 00:52:58,542] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 869.89 | bwd: 3232.22 | bwd_inner: 3213.75 | bwd_allreduce: 18.43 | step: 191.58

 84%|████████▍ | 76/90 [08:08<01:03,  4.53s/it]
                                               
{'loss': 0.0616, 'learning_rate': 1.2508867458185037e-06, 'epoch': 2.53}

 84%|████████▍ | 76/90 [08:08<01:03,  4.53s/it]dynamic ViT batch size: 544, images per sample: 544.0, dynamic token length: 2983
[2026-02-24 00:53:00,082] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 788.93 | bwd_microstep: 691.00 | bwd_inner_microstep: 690.98 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.06
dynamic ViT batch size: 276, images per sample: 276.0, dynamic token length: 2715
[2026-02-24 00:53:01,277] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.46 | optimizer_gradients: 9.71 | optimizer_step: 9.90
[2026-02-24 00:53:01,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 330.69 | bwd_microstep: 754.26 | bwd_inner_microstep: 712.97 | bwd_allreduce_microstep: 41.25 | step_microstep: 91.22
[2026-02-24 00:53:01,278] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 1119.62 | bwd: 1445.28 | bwd_inner: 1403.95 | bwd_allreduce: 41.27 | step: 91.29

 86%|████████▌ | 77/90 [08:10<00:51,  3.99s/it]
                                               
{'loss': 0.1043, 'learning_rate': 1.0817524278461777e-06, 'epoch': 2.57}

 86%|████████▌ | 77/90 [08:10<00:51,  3.99s/it]dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2979
[2026-02-24 00:53:07,559] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 545.23 | bwd_microstep: 693.79 | bwd_inner_microstep: 693.76 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.03
dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2715
[2026-02-24 00:53:08,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 59.63 | optimizer_gradients: 9.68 | optimizer_step: 9.89
[2026-02-24 00:53:08,968] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 533.41 | bwd_microstep: 750.72 | bwd_inner_microstep: 733.68 | bwd_allreduce_microstep: 17.00 | step_microstep: 93.93
[2026-02-24 00:53:08,969] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 1078.65 | bwd: 1444.52 | bwd_inner: 1427.44 | bwd_allreduce: 17.02 | step: 93.97

 87%|████████▋ | 78/90 [08:18<01:01,  5.10s/it]
                                               
{'loss': 0.0307, 'learning_rate': 9.242458032904311e-07, 'epoch': 2.6}

 87%|████████▋ | 78/90 [08:18<01:01,  5.10s/it]dynamic ViT batch size: 536, images per sample: 536.0, dynamic token length: 2711
[2026-02-24 00:53:10,198] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 529.71 | bwd_microstep: 659.05 | bwd_inner_microstep: 659.03 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.04
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:53:11,644] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 57.83 | optimizer_gradients: 10.41 | optimizer_step: 9.91
[2026-02-24 00:53:11,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.02 | bwd_microstep: 1014.39 | bwd_inner_microstep: 972.84 | bwd_allreduce_microstep: 41.35 | step_microstep: 90.41
[2026-02-24 00:53:11,645] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 852.73 | bwd: 1673.45 | bwd_inner: 1631.89 | bwd_allreduce: 41.36 | step: 90.46

 88%|████████▊ | 79/90 [08:21<00:48,  4.37s/it]
                                               
{'loss': 0.036, 'learning_rate': 7.785722307406685e-07, 'epoch': 2.63}

 88%|████████▊ | 79/90 [08:21<00:48,  4.37s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:53:12,495] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.91 | bwd_microstep: 502.28 | bwd_inner_microstep: 502.26 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.05
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:53:21,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.43 | optimizer_gradients: 9.65 | optimizer_step: 9.88
[2026-02-24 00:53:21,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.37 | bwd_microstep: 8479.29 | bwd_inner_microstep: 8435.39 | bwd_allreduce_microstep: 43.82 | step_microstep: 90.73
[2026-02-24 00:53:21,409] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 648.28 | bwd: 8981.58 | bwd_inner: 8937.67 | bwd_allreduce: 43.81 | step: 90.85

 89%|████████▉ | 80/90 [08:30<00:59,  5.99s/it]
                                               
{'loss': 0.106, 'learning_rate': 6.449216407439906e-07, 'epoch': 2.67}

 89%|████████▉ | 80/90 [08:30<00:59,  5.99s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:53:22,264] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.83 | bwd_microstep: 503.14 | bwd_inner_microstep: 503.12 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.04
dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2715
[2026-02-24 00:53:24,132] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.07 | optimizer_gradients: 9.69 | optimizer_step: 9.88
[2026-02-24 00:53:24,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 533.98 | bwd_microstep: 1213.51 | bwd_inner_microstep: 1198.42 | bwd_allreduce_microstep: 15.05 | step_microstep: 89.86
[2026-02-24 00:53:24,133] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 857.81 | bwd: 1716.66 | bwd_inner: 1701.54 | bwd_allreduce: 15.07 | step: 89.91

 90%|█████████ | 81/90 [08:33<00:45,  5.01s/it]
                                               
{'loss': 0.044, 'learning_rate': 5.234682881719766e-07, 'epoch': 2.7}

 90%|█████████ | 81/90 [08:33<00:45,  5.01s/it]dynamic ViT batch size: 288, images per sample: 288.0, dynamic token length: 2727
[2026-02-24 00:53:25,015] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 340.14 | bwd_microstep: 516.74 | bwd_inner_microstep: 516.72 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.05
dynamic ViT batch size: 484, images per sample: 484.0, dynamic token length: 2659
[2026-02-24 00:53:26,726] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.37 | optimizer_gradients: 9.71 | optimizer_step: 9.88
[2026-02-24 00:53:26,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 489.32 | bwd_microstep: 1101.79 | bwd_inner_microstep: 1059.79 | bwd_allreduce_microstep: 41.91 | step_microstep: 91.39
[2026-02-24 00:53:26,727] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 829.46 | bwd: 1618.54 | bwd_inner: 1576.53 | bwd_allreduce: 41.92 | step: 91.45

 91%|█████████ | 82/90 [08:36<00:34,  4.29s/it]
                                               
{'loss': 0.0273, 'learning_rate': 4.1437052502558693e-07, 'epoch': 2.73}

 91%|█████████ | 82/90 [08:36<00:34,  4.29s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:53:27,583] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 323.56 | bwd_microstep: 503.40 | bwd_inner_microstep: 503.38 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.05
dynamic ViT batch size: 248, images per sample: 248.0, dynamic token length: 2423
[2026-02-24 00:53:31,407] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.45 | optimizer_gradients: 9.64 | optimizer_step: 9.89
[2026-02-24 00:53:31,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 293.23 | bwd_microstep: 3420.24 | bwd_inner_microstep: 3379.47 | bwd_allreduce_microstep: 40.56 | step_microstep: 92.83
[2026-02-24 00:53:31,408] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 616.77 | bwd: 3923.65 | bwd_inner: 3882.88 | bwd_allreduce: 40.57 | step: 92.90

 92%|█████████▏| 83/90 [08:40<00:30,  4.40s/it]
                                               
{'loss': 0.0534, 'learning_rate': 3.1777059397436693e-07, 'epoch': 2.77}

 92%|█████████▏| 83/90 [08:40<00:30,  4.40s/it]dynamic ViT batch size: 480, images per sample: 480.0, dynamic token length: 2655
[2026-02-24 00:53:32,560] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 485.45 | bwd_microstep: 631.05 | bwd_inner_microstep: 631.03 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.10
dynamic ViT batch size: 272, images per sample: 272.0, dynamic token length: 2711
[2026-02-24 00:53:34,137] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.32 | optimizer_gradients: 9.61 | optimizer_step: 9.87
[2026-02-24 00:53:34,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 327.22 | bwd_microstep: 1140.38 | bwd_inner_microstep: 1096.28 | bwd_allreduce_microstep: 43.86 | step_microstep: 90.36
[2026-02-24 00:53:34,138] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 812.67 | bwd: 1771.44 | bwd_inner: 1727.50 | bwd_allreduce: 43.87 | step: 90.47

 93%|█████████▎| 84/90 [08:43<00:23,  3.90s/it]
                                               
{'loss': 0.0654, 'learning_rate': 2.3379444289913344e-07, 'epoch': 2.8}

 93%|█████████▎| 84/90 [08:43<00:23,  3.90s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:53:34,993] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 324.47 | bwd_microstep: 503.40 | bwd_inner_microstep: 503.37 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.07
dynamic ViT batch size: 288, images per sample: 288.0, dynamic token length: 2727
[2026-02-24 00:53:41,515] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.24 | optimizer_gradients: 9.68 | optimizer_step: 9.87
[2026-02-24 00:53:41,516] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 339.01 | bwd_microstep: 2410.73 | bwd_inner_microstep: 2367.06 | bwd_allreduce_microstep: 43.62 | step_microstep: 90.91
[2026-02-24 00:53:41,517] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 663.48 | bwd: 2914.13 | bwd_inner: 2870.43 | bwd_allreduce: 43.63 | step: 91.00

 94%|█████████▍| 85/90 [08:51<00:24,  4.94s/it]
                                               
{'loss': 0.0625, 'learning_rate': 1.6255156067997325e-07, 'epoch': 2.83}

 94%|█████████▍| 85/90 [08:51<00:24,  4.94s/it]dynamic ViT batch size: 540, images per sample: 540.0, dynamic token length: 2715
[2026-02-24 00:53:42,753] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 533.88 | bwd_microstep: 661.20 | bwd_inner_microstep: 661.17 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.07
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:53:47,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.44 | optimizer_gradients: 9.63 | optimizer_step: 9.86
[2026-02-24 00:53:47,289] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.12 | bwd_microstep: 4103.90 | bwd_inner_microstep: 4060.62 | bwd_allreduce_microstep: 43.24 | step_microstep: 90.72
[2026-02-24 00:53:47,290] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 856.00 | bwd: 4765.10 | bwd_inner: 4721.79 | bwd_allreduce: 43.25 | step: 90.80

 96%|█████████▌| 86/90 [08:56<00:20,  5.19s/it]
                                               
{'loss': 0.0373, 'learning_rate': 1.041348344436277e-07, 'epoch': 2.87}

 96%|█████████▌| 86/90 [08:56<00:20,  5.19s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:53:48,139] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.52 | bwd_microstep: 501.02 | bwd_inner_microstep: 500.99 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.06
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:53:50,017] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.40 | optimizer_gradients: 9.54 | optimizer_step: 9.89
[2026-02-24 00:53:50,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.48 | bwd_microstep: 1445.59 | bwd_inner_microstep: 1401.21 | bwd_allreduce_microstep: 44.35 | step_microstep: 91.40
[2026-02-24 00:53:50,018] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 644.99 | bwd: 1946.61 | bwd_inner: 1902.20 | bwd_allreduce: 44.26 | step: 91.46

 97%|█████████▋| 87/90 [08:59<00:13,  4.45s/it]
                                               
{'loss': 0.0372, 'learning_rate': 5.862042845640403e-08, 'epoch': 2.9}

 97%|█████████▋| 87/90 [08:59<00:13,  4.45s/it]dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
[2026-02-24 00:53:50,867] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 322.21 | bwd_microstep: 502.19 | bwd_inner_microstep: 502.17 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.03
dynamic ViT batch size: 256, images per sample: 256.0, dynamic token length: 2431
[2026-02-24 00:53:52,452] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.21 | optimizer_gradients: 9.56 | optimizer_step: 9.91
[2026-02-24 00:53:52,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 299.55 | bwd_microstep: 1179.01 | bwd_inner_microstep: 1137.88 | bwd_allreduce_microstep: 41.07 | step_microstep: 89.38
[2026-02-24 00:53:52,453] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 621.76 | bwd: 1681.21 | bwd_inner: 1640.06 | bwd_allreduce: 41.08 | step: 89.41

 98%|█████████▊| 88/90 [09:01<00:07,  3.85s/it]
                                               
{'loss': 0.0709, 'learning_rate': 2.606768482050215e-08, 'epoch': 2.93}

 98%|█████████▊| 88/90 [09:01<00:07,  3.85s/it]dynamic ViT batch size: 252, images per sample: 252.0, dynamic token length: 2427
[2026-02-24 00:53:53,223] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 295.77 | bwd_microstep: 452.07 | bwd_inner_microstep: 452.04 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.03
dynamic ViT batch size: 544, images per sample: 544.0, dynamic token length: 2983
[2026-02-24 00:53:55,878] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.87 | optimizer_gradients: 9.58 | optimizer_step: 9.90
[2026-02-24 00:53:55,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 544.17 | bwd_microstep: 777.00 | bwd_inner_microstep: 768.92 | bwd_allreduce_microstep: 8.03 | step_microstep: 97.14
[2026-02-24 00:53:55,879] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 839.94 | bwd: 1229.07 | bwd_inner: 1220.97 | bwd_allreduce: 8.04 | step: 97.18

 99%|█████████▉| 89/90 [09:05<00:03,  3.72s/it]
                                               
{'loss': 0.0299, 'learning_rate': 6.5190461032305085e-09, 'epoch': 2.97}

 99%|█████████▉| 89/90 [09:05<00:03,  3.72s/it]dynamic ViT batch size: 536, images per sample: 536.0, dynamic token length: 2711
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
[2026-02-24 00:53:57,988] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 994.36 | bwd_microstep: 1078.53 | bwd_inner_microstep: 1078.51 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.04
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
dynamic ViT batch size: 268, images per sample: 268.0, dynamic token length: 2707
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /root/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
[2026-02-24 00:54:02,595] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | optimizer_allgather: 58.32 | optimizer_gradients: 9.54 | optimizer_step: 9.88
[2026-02-24 00:54:02,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd_microstep: 321.37 | bwd_microstep: 3021.32 | bwd_inner_microstep: 2977.23 | bwd_allreduce_microstep: 44.01 | step_microstep: 90.24
[2026-02-24 00:54:02,596] [INFO] [logging.py:128:log_dist] [Rank 0] time (ms) | fwd: 1315.73 | bwd: 4099.86 | bwd_inner: 4055.77 | bwd_allreduce: 43.93 | step: 90.30

100%|██████████| 90/90 [09:12<00:00,  4.62s/it]
                                               
{'loss': 0.0781, 'learning_rate': 0.0, 'epoch': 3.0}

100%|██████████| 90/90 [09:12<00:00,  4.62s/it][INFO|trainer.py:1962] 2026-02-24 00:54:02,607 >> 

Training completed. Do not forget to share your model on huggingface.co/models =)



                                               
{'train_runtime': 552.1008, 'train_samples_per_second': 2.597, 'train_steps_per_second': 0.163, 'train_loss': 0.2259764001187351, 'epoch': 3.0}

100%|██████████| 90/90 [09:12<00:00,  4.62s/it]
100%|██████████| 90/90 [09:12<00:00,  6.13s/it]
[INFO|trainer.py:2936] 2026-02-24 00:54:10,888 >> Saving model checkpoint to /mnt/shared-storage-user/jiaziheng/LMMS/internvl-pretrain-02_23_full-yt-gaming-8B
[INFO|configuration_utils.py:473] 2026-02-24 00:54:10,892 >> Configuration saved in /mnt/shared-storage-user/jiaziheng/LMMS/internvl-pretrain-02_23_full-yt-gaming-8B/config.json
[INFO|configuration_utils.py:594] 2026-02-24 00:54:10,893 >> Configuration saved in /mnt/shared-storage-user/jiaziheng/LMMS/internvl-pretrain-02_23_full-yt-gaming-8B/generation_config.json
[INFO|modeling_utils.py:2501] 2026-02-24 00:54:25,510 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 4 checkpoint shards. You can find where each parameters has been saved in the index located at /mnt/shared-storage-user/jiaziheng/LMMS/internvl-pretrain-02_23_full-yt-gaming-8B/model.safetensors.index.json.
[INFO|tokenization_utils_base.py:2433] 2026-02-24 00:54:25,511 >> tokenizer config file saved in /mnt/shared-storage-user/jiaziheng/LMMS/internvl-pretrain-02_23_full-yt-gaming-8B/tokenizer_config.json
[INFO|tokenization_utils_base.py:2442] 2026-02-24 00:54:25,512 >> Special tokens file saved in /mnt/shared-storage-user/jiaziheng/LMMS/internvl-pretrain-02_23_full-yt-gaming-8B/special_tokens_map.json
[INFO|tokenization_utils_base.py:2493] 2026-02-24 00:54:25,512 >> added tokens file saved in /mnt/shared-storage-user/jiaziheng/LMMS/internvl-pretrain-02_23_full-yt-gaming-8B/added_tokens.json
***** train metrics *****
  epoch                    =        3.0
  train_loss               =      0.226
  train_runtime            = 0:09:12.10
  train_samples            =        478
  train_samples_per_second =      2.597
  train_steps_per_second   =      0.163
[rank0]:[W224 00:54:26.294996144 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
W0224 01:11:19.194000 102520 site-packages/torch/distributed/run.py:792] 
W0224 01:11:19.194000 102520 site-packages/torch/distributed/run.py:792] *****************************************
W0224 01:11:19.194000 102520 site-packages/torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0224 01:11:19.194000 102520 site-packages/torch/distributed/run.py:792] *****************************************