0: W1123 14:35:36.265000 2356243 torch/distributed/run.py:792] 
0: W1123 14:35:36.265000 2356243 torch/distributed/run.py:792] *****************************************
0: W1123 14:35:36.265000 2356243 torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
0: W1123 14:35:36.265000 2356243 torch/distributed/run.py:792] *****************************************
3: W1123 14:35:36.265000 51501 torch/distributed/run.py:792] 
3: W1123 14:35:36.265000 51501 torch/distributed/run.py:792] *****************************************
3: W1123 14:35:36.265000 51501 torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
3: W1123 14:35:36.265000 51501 torch/distributed/run.py:792] *****************************************
2: W1123 14:35:36.265000 895971 torch/distributed/run.py:792] 
2: W1123 14:35:36.265000 895971 torch/distributed/run.py:792] *****************************************
2: W1123 14:35:36.265000 895971 torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
2: W1123 14:35:36.265000 895971 torch/distributed/run.py:792] *****************************************
1: W1123 14:35:36.265000 2965489 torch/distributed/run.py:792] 
1: W1123 14:35:36.265000 2965489 torch/distributed/run.py:792] *****************************************
1: W1123 14:35:36.265000 2965489 torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
1: W1123 14:35:36.265000 2965489 torch/distributed/run.py:792] *****************************************
2: [2025-11-23 14:40:32,705] [INFO] [axolotl.utils.schemas.validation.check_eval_packing:119] [PID:896064] [RANK:0] explicitly setting `eval_sample_packing` to match `sample_packing`
3: [2025-11-23 14:40:32,705] [INFO] [axolotl.utils.schemas.validation.check_eval_packing:119] [PID:51582] [RANK:0] explicitly setting `eval_sample_packing` to match `sample_packing`
2: [2025-11-23 14:40:32,705] [INFO] [axolotl.utils.schemas.validation.hint_sample_packing_padding:218] [PID:896064] [RANK:0] Setting `pad_to_sequence_len: true` to prevent memory leaks when sample_packing
3: [2025-11-23 14:40:32,705] [INFO] [axolotl.utils.schemas.validation.hint_sample_packing_padding:218] [PID:51582] [RANK:0] Setting `pad_to_sequence_len: true` to prevent memory leaks when sample_packing
0: [2025-11-23 14:40:32,706] [INFO] [axolotl.utils.schemas.validation.check_eval_packing:119] [PID:2356327] [RANK:0] explicitly setting `eval_sample_packing` to match `sample_packing`
1: [2025-11-23 14:40:32,705] [INFO] [axolotl.utils.schemas.validation.check_eval_packing:119] [PID:2965582] [RANK:0] explicitly setting `eval_sample_packing` to match `sample_packing`
0: [2025-11-23 14:40:32,706] [INFO] [axolotl.utils.schemas.validation.hint_sample_packing_padding:218] [PID:2356327] [RANK:0] Setting `pad_to_sequence_len: true` to prevent memory leaks when sample_packing
1: [2025-11-23 14:40:32,706] [INFO] [axolotl.utils.schemas.validation.hint_sample_packing_padding:218] [PID:2965582] [RANK:0] Setting `pad_to_sequence_len: true` to prevent memory leaks when sample_packing
0: [2025-11-23 14:41:03,121] [WARNING] [axolotl.utils.config.normalize_config:139] [PID:2356327] [RANK:0] Invalid value for save_steps (1.6666666666666667) from saves_per_epoch and/or num_epochs. Saving at training end only.
0: [2025-11-23 14:41:03,271] [INFO] [axolotl.cli.config.load_cfg:245] [PID:2356327] [RANK:0] config:
0: {
0:   "activation_offloading": false,
0:   "auto_resume_from_checkpoints": true,
0:   "axolotl_config_path": "/lustre/fswork/projects/rech/dgo/udv55np/train/tmp/1763904854732780523.yaml",
0:   "base_model": "/lustre/fswork/projects/rech/qwv/udv55np/Gemma/base/gemma-3-4b",
0:   "base_model_config": "/lustre/fswork/projects/rech/qwv/udv55np/Gemma/base/gemma-3-4b",
0:   "batch_size": 16,
0:   "bf16": true,
0:   "capabilities": {
0:     "bf16": true,
0:     "compute_capability": "sm_90",
0:     "fp8": false,
0:     "n_gpu": 16,
0:     "n_node": 1
0:   },
0:   "chat_template": "gemma3",
0:   "context_parallel_size": 1,
0:   "curriculum_sampling": true,
0:   "dataloader_num_workers": 2,
0:   "dataset_prepared_path": "/lustre/fswork/projects/rech/dgo/udv55np/dataset_gemma/Nemotron-Super-49B-v1_5/split_0.75",
0:   "dataset_processes": 32,
0:   "datasets": [
0:     {
0:       "chat_template": "tokenizer_default",
0:       "data_files": [
0:         "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0004.jsonl",
0:         "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0011.jsonl",
0:         "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0000.jsonl",
0:         "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0003.jsonl"
0:       ],
0:       "ds_type": "json",
0:       "field_messages": "conversations",
0:       "message_property_mappings": {
0:         "content": "content",
0:         "role": "role"
0:       },
0:       "path": "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking",
0:       "trust_remote_code": false,
0:       "type": "chat_template"
0:     },
0:     {
0:       "chat_template": "tokenizer_default",
0:       "data_files": [
0:         "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0007.jsonl",
0:         "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0009.jsonl",
0:         "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0005.jsonl",
0:         "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0006.jsonl",
0:         "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0014.jsonl",
0:         "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0010.jsonl",
0:         "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0012.jsonl",
0:         "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0008.jsonl",
0:         "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0001.jsonl",
0:         "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0002.jsonl",
0:         "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0013.jsonl",
0:         "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking/0015.jsonl"
0:       ],
0:       "ds_type": "json",
0:       "field_messages": "conversations",
0:       "message_property_mappings": {
0:         "content": "content",
0:         "role": "role"
0:       },
0:       "path": "/lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking",
0:       "trust_remote_code": false,
0:       "type": "chat_template"
0:     }
0:   ],
0:   "ddp": true,
0:   "deepspeed": {
0:     "bf16": {
0:       "enabled": true
0:     },
0:     "gradient_accumulation_steps": "auto",
0:     "gradient_clipping": "auto",
0:     "train_batch_size": "auto",
0:     "train_micro_batch_size_per_gpu": "auto",
0:     "wall_clock_breakdown": false,
0:     "zero_optimization": {
0:       "contiguous_gradients": true,
0:       "overlap_comm": true,
0:       "reduce_bucket_size": "auto",
0:       "stage": 3,
0:       "stage3_gather_16bit_weights_on_model_save": true,
0:       "stage3_param_persistence_threshold": "auto",
0:       "stage3_prefetch_bucket_size": "auto",
0:       "sub_group_size": 0
0:     }
0:   },
0:   "device": "cuda:0",
0:   "device_map": {
0:     "": 0
0:   },
0:   "dion_rank_fraction": 1.0,
0:   "dion_rank_multiple_of": 1,
0:   "env_capabilities": {
0:     "torch_version": "2.6.0"
0:   },
0:   "eot_tokens": [
0:     "<end_of_turn>"
0:   ],
0:   "eval_batch_size": 1,
0:   "eval_causal_lm_metrics": [
0:     "sacrebleu",
0:     "comet",
0:     "ter",
0:     "chrf"
0:   ],
0:   "eval_max_new_tokens": 128,
0:   "eval_sample_packing": true,
0:   "eval_table_size": 0,
0:   "evals_per_epoch": 0,
0:   "flash_attention": true,
0:   "fp16": false,
0:   "gradient_accumulation_steps": 1,
0:   "gradient_checkpointing": true,
0:   "gradient_checkpointing_kwargs": {
0:     "use_reentrant": true
0:   },
0:   "is_multimodal": true,
0:   "learning_rate": 5e-06,
0:   "lisa_layers_attribute": "model.layers",
0:   "load_best_model_at_end": false,
0:   "load_in_4bit": false,
0:   "load_in_8bit": false,
0:   "local_rank": 0,
0:   "logging_steps": 10,
0:   "lora_dropout": 0.0,
0:   "loraplus_lr_embedding": 1e-06,
0:   "lr_scheduler": "warmup_stable_decay",
0:   "lr_scheduler_kwargs": {
0:     "min_lr_ratio": 0.1,
0:     "num_decay_steps": 200
0:   },
0:   "max_prompt_len": 512,
0:   "mean_resizing_embeddings": false,
0:   "micro_batch_size": 1,
0:   "model_config_type": "gemma3",
0:   "num_epochs": 0.6,
0:   "optimizer": "adamw_torch_fused",
0:   "output_dir": "/lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-4b/0.75",
0:   "pad_to_sequence_len": true,
0:   "pretrain_multipack_attn": true,
0:   "pretrain_multipack_buffer_size": 10000,
0:   "processor_config": "/lustre/fswork/projects/rech/qwv/udv55np/Gemma/base/gemma-3-4b",
0:   "profiler_steps_start": 0,
0:   "qlora_sharded_model_loading": false,
0:   "ray_num_workers": 1,
0:   "resources_per_worker": {
0:     "GPU": 1
0:   },
0:   "sample_packing": true,
0:   "sample_packing_bin_size": 200,
0:   "sample_packing_group_size": 100000,
0:   "sample_packing_sequentially": true,
0:   "save_only_model": true,
0:   "save_safetensors": true,
0:   "save_total_limit": 20,
0:   "saves_per_epoch": 1,
0:   "sequence_len": 16384,
0:   "shuffle_before_merging_datasets": true,
0:   "shuffle_merged_datasets": false,
0:   "skip_prepare_dataset": false,
0:   "strict": false,
0:   "tensor_parallel_size": 1,
0:   "tf32": false,
0:   "tiled_mlp_use_original_mlp": true,
0:   "tokenizer_config": "/lustre/fswork/projects/rech/qwv/udv55np/Gemma/base/gemma-3-27b",
0:   "torch_dtype": "torch.bfloat16",
0:   "train_on_inputs": false,
0:   "trl": {
0:     "log_completions": false,
0:     "mask_truncated_completions": false,
0:     "ref_model_mixup_alpha": 0.9,
0:     "ref_model_sync_steps": 64,
0:     "scale_rewards": true,
0:     "sync_ref_model": false,
0:     "use_vllm": false,
0:     "vllm_server_host": "0.0.0.0",
0:     "vllm_server_port": 8000
0:   },
0:   "use_ray": false,
0:   "use_tensorboard": true,
0:   "val_set_size": 0.0,
0:   "vllm": {
0:     "device": "auto",
0:     "dtype": "auto",
0:     "gpu_memory_utilization": 0.9,
0:     "host": "0.0.0.0",
0:     "port": 8000
0:   },
0:   "warmup_steps": 100,
0:   "weight_decay": 0.0,
0:   "world_size": 16
0: }
0: [2025-11-23 14:41:03,273] [INFO] [axolotl.cli.checks.check_user_token:35] [PID:2356327] [RANK:0] Skipping HuggingFace token verification because HF_HUB_OFFLINE is set to True. Only local files will be used.
1: [2025-11-23 14:41:05,205] [INFO] [axolotl.utils.data.sft._load_raw_datasets:314] [PID:2965584] [RANK:2] Loading raw datasets...
1: 
Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 4677 examples [00:00, 11780.35 examples/s]
Generating train split: 9505 examples [00:00, 21404.17 examples/s]
Generating train split: 28019 examples [00:00, 59254.09 examples/s]
Generating train split: 39485 examples [00:00, 62880.71 examples/s]
Generating train split: 53359 examples [00:00, 70474.35 examples/s]
Generating train split: 69660 examples [00:01, 87071.54 examples/s]
Generating train split: 88600 examples [00:01, 79726.65 examples/s]
Generating train split: 104490 examples [00:01, 91292.05 examples/s]
Generating train split: 123247 examples [00:01, 84840.27 examples/s]
Generating train split: 139320 examples [00:01, 95490.75 examples/s]
Generating train split: 139320 examples [00:01, 75180.64 examples/s]
1: [2025-11-23 14:41:09,428] [INFO] [axolotl.utils.data.wrappers.get_dataset_wrapper:88] [PID:2965584] [RANK:2] Loading dataset: /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking with base_type: chat_template and prompt_style: None
1: 
Tokenizing Prompts (num_proc=32):   0%|          | 0/139320 [00:00<?, ? examples/s]
Tokenizing Prompts (num_proc=32):   1%|          | 1000/139320 [00:04<10:12, 225.65 examples/s]
Tokenizing Prompts (num_proc=32):   1%|▏         | 2000/139320 [00:04<04:30, 507.14 examples/s]
Tokenizing Prompts (num_proc=32):   3%|▎         | 4000/139320 [00:05<02:00, 1121.82 examples/s]
Tokenizing Prompts (num_proc=32):   4%|▎         | 5000/139320 [00:05<01:33, 1431.20 examples/s]
Tokenizing Prompts (num_proc=32):   4%|▍         | 6000/139320 [00:05<01:15, 1771.07 examples/s]
Tokenizing Prompts (num_proc=32):   6%|▌         | 8000/139320 [00:06<00:55, 2348.00 examples/s]
Tokenizing Prompts (num_proc=32):   6%|▋         | 9000/139320 [00:06<00:50, 2570.52 examples/s]
Tokenizing Prompts (num_proc=32):   8%|▊         | 11000/139320 [00:06<00:35, 3626.92 examples/s]
Tokenizing Prompts (num_proc=32):   9%|▊         | 12000/139320 [00:07<00:35, 3611.48 examples/s]
Tokenizing Prompts (num_proc=32):   9%|▉        
1:  | 13000/139320 [00:07<00:33, 3729.44 examples/s]
Tokenizing Prompts (num_proc=32):  11%|█         | 15000/139320 [00:07<00:26, 4698.24 examples/s]
Tokenizing Prompts (num_proc=32):  12%|█▏        | 17000/139320 [00:07<00:22, 5465.30 examples/s]
Tokenizing Prompts (num_proc=32):  14%|█▎        | 19000/139320 [00:08<00:19, 6064.88 examples/s]
Tokenizing Prompts (num_proc=32):  14%|█▍        | 20000/139320 [00:08<00:22, 5394.70 examples/s]
Tokenizing Prompts (num_proc=32):  17%|█▋        | 23000/139320 [00:08<00:16, 7083.07 examples/s]
Tokenizing Prompts (num_proc=32):  18%|█▊        | 25000/139320 [00:08<00:14, 7666.37 examples/s]
Tokenizing Prompts (num_proc=32):  19%|█▊        | 26000/139320 [00:09<00:15, 7167.69 examples/s]
Tokenizing Prompts (num_proc=32):  20%|██        | 28000/139320 [00:09<00:12, 8614.25 examples/s]
Tokenizing Prompts (num_proc=32):  22%|██▏       | 31000/139320 [00:09<00:11, 9410.34 examples/s]
Tokenizing Prompts (num_proc=32):  24%|██▎       |
1:  33000/139320 [00:09<00:14, 7324.10 examples/s]
Tokenizing Prompts (num_proc=32):  24%|██▍       | 34000/139320 [00:09<00:13, 7542.05 examples/s]
Tokenizing Prompts (num_proc=32):  26%|██▌       | 36000/139320 [00:10<00:11, 9099.01 examples/s]
Tokenizing Prompts (num_proc=32):  29%|██▊       | 40000/139320 [00:10<00:07, 12600.39 examples/s]
Tokenizing Prompts (num_proc=32):  30%|███       | 42000/139320 [00:10<00:08, 11584.07 examples/s]
Tokenizing Prompts (num_proc=32):  32%|███▏      | 44000/139320 [00:10<00:08, 11152.04 examples/s]
Tokenizing Prompts (num_proc=32):  33%|███▎      | 46000/139320 [00:11<00:10, 8703.38 examples/s] 
Tokenizing Prompts (num_proc=32):  34%|███▎      | 47000/139320 [00:11<00:10, 8624.79 examples/s]
Tokenizing Prompts (num_proc=32):  37%|███▋      | 51000/139320 [00:11<00:06, 12811.46 examples/s]
Tokenizing Prompts (num_proc=32):  38%|███▊      | 53000/139320 [00:11<00:06, 13302.55 examples/s]
Tokenizing Prompts (num_
1: proc=32):  39%|███▉      | 55000/139320 [00:11<00:07, 11042.38 examples/s]
Tokenizing Prompts (num_proc=32):  41%|████      | 57000/139320 [00:11<00:07, 10835.19 examples/s]
Tokenizing Prompts (num_proc=32):  42%|████▏     | 59000/139320 [00:12<00:06, 12427.86 examples/s]
Tokenizing Prompts (num_proc=32):  46%|████▌     | 64000/139320 [00:12<00:05, 14740.77 examples/s]
Tokenizing Prompts (num_proc=32):  47%|████▋     | 66000/139320 [00:12<00:08, 9150.86 examples/s] 
Tokenizing Prompts (num_proc=32):  50%|████▉     | 69000/139320 [00:12<00:06, 11337.62 examples/s]
Tokenizing Prompts (num_proc=32):  51%|█████     | 71000/139320 [00:13<00:05, 11423.54 examples/s]
Tokenizing Prompts (num_proc=32):  52%|█████▏    | 73000/139320 [00:13<00:06, 9545.89 examples/s] 
Tokenizing Prompts (num_proc=32):  53%|█████▎    | 74354/139320 [00:13<00:06, 9605.50 examples/s]
Tokenizing Prompts (num_proc=32):  56%|█████▌    | 77354
1: /139320 [00:13<00:06, 10140.08 examples/s]
Tokenizing Prompts (num_proc=32):  58%|█████▊    | 81062/139320 [00:13<00:04, 12883.42 examples/s]
Tokenizing Prompts (num_proc=32):  60%|██████    | 84062/139320 [00:14<00:03, 14313.00 examples/s]
Tokenizing Prompts (num_proc=32):  62%|██████▏   | 87062/139320 [00:14<00:04, 12039.18 examples/s]
Tokenizing Prompts (num_proc=32):  64%|██████▍   | 89416/139320 [00:14<00:03, 12939.91 examples/s]
Tokenizing Prompts (num_proc=32):  66%|██████▋   | 92416/139320 [00:14<00:03, 15199.86 examples/s]
Tokenizing Prompts (num_proc=32):  68%|██████▊   | 94416/139320 [00:14<00:02, 15460.32 examples/s]
Tokenizing Prompts (num_proc=32):  70%|███████   | 97770/139320 [00:15<00:02, 17813.50 examples/s]
Tokenizing Prompts (num_proc=32):  72%|███████▏  | 100478/139320 [00:15<00:04, 8661.96 examples/s]
Tokenizing Prompts (num_proc=32):  73%|███████▎  | 102186/13932
1: 0 [00:15<00:04, 8086.88 examples/s]
Tokenizing Prompts (num_proc=32):  75%|███████▍  | 103894/139320 [00:16<00:03, 9135.58 examples/s]
Tokenizing Prompts (num_proc=32):  76%|███████▌  | 105894/139320 [00:16<00:03, 8854.86 examples/s]
Tokenizing Prompts (num_proc=32):  78%|███████▊  | 108894/139320 [00:16<00:02, 11764.78 examples/s]
Tokenizing Prompts (num_proc=32):  80%|███████▉  | 111248/139320 [00:16<00:02, 11160.99 examples/s]
Tokenizing Prompts (num_proc=32):  81%|████████▏ | 113248/139320 [00:16<00:02, 10013.26 examples/s]
Tokenizing Prompts (num_proc=32):  82%|████████▏ | 114602/139320 [00:17<00:02, 9229.84 examples/s] 
Tokenizing Prompts (num_proc=32):  84%|████████▍ | 117602/139320 [00:17<00:02, 10236.17 examples/s]
Tokenizing Prompts (num_proc=32):  87%|████████▋ | 120956/139320 [00:17<00:01, 11400.34 examples/s]
Tokenizing Prompts (num_proc=32):  88%|█████
1: ███▊ | 122664/139320 [00:17<00:01, 11355.61 examples/s]
Tokenizing Prompts (num_proc=32):  89%|████████▉ | 124080/139320 [00:18<00:02, 6319.61 examples/s] 
Tokenizing Prompts (num_proc=32):  90%|████████▉ | 125080/139320 [00:18<00:02, 6639.83 examples/s]
Tokenizing Prompts (num_proc=32):  90%|█████████ | 126080/139320 [00:18<00:01, 6956.31 examples/s]
Tokenizing Prompts (num_proc=32):  91%|█████████ | 127080/139320 [00:18<00:01, 7245.46 examples/s]
Tokenizing Prompts (num_proc=32):  92%|█████████▏| 128434/139320 [00:18<00:01, 7265.46 examples/s]
Tokenizing Prompts (num_proc=32):  93%|█████████▎| 129434/139320 [00:19<00:01, 7299.73 examples/s]
Tokenizing Prompts (num_proc=32):  94%|█████████▎| 130434/139320 [00:19<00:01, 5569.74 examples/s]
Tokenizing Prompts (num_proc=32):  94%|█████████▍| 131434/139320 [00:19<00:01, 5410.54 examples/s]
Tokenizing Prom
1: pts (num_proc=32):  95%|█████████▍| 132142/139320 [00:19<00:01, 5667.65 examples/s]
Tokenizing Prompts (num_proc=32):  96%|█████████▌| 133496/139320 [00:19<00:01, 5548.87 examples/s]
Tokenizing Prompts (num_proc=32):  97%|█████████▋| 135202/139320 [00:20<00:00, 6950.40 examples/s]
Tokenizing Prompts (num_proc=32):  98%|█████████▊| 137202/139320 [00:20<00:00, 8318.32 examples/s]
Tokenizing Prompts (num_proc=32):  99%|█████████▉| 138261/139320 [00:20<00:00, 4264.99 examples/s]
Tokenizing Prompts (num_proc=32): 100%|█████████▉| 138967/139320 [00:20<00:00, 4471.76 examples/s]
Tokenizing Prompts (num_proc=32): 100%|██████████| 139320/139320 [00:21<00:00, 6537.44 examples/s]
1: 
Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1048 examples [00:00, 7827.60 examples/s]
Generating train split: 4077 examples [00:00, 14871.13 examples/s]
Generating train split: 8157 examples [00:00, 19760.15 examples/s]
Generating train split: 12358 examples [00:00, 24219.10 examples/s]
Generating train split: 16534 examples [00:00, 27080.25 examples/s]
Generating train split: 20554 examples [00:00, 28646.79 examples/s]
Generating train split: 24725 examples [00:00, 29997.79 examples/s]
Generating train split: 28893 examples [00:01, 30985.06 examples/s]
Generating train split: 33101 examples [00:01, 31938.77 examples/s]
Generating train split: 36958 examples [00:01, 23559.66 examples/s]
Generating train split: 41160 examples [00:01, 24495.04 examples/s]
Generating train split: 45340 examples [00:01, 25629.97 examples/s]
Generating train split: 49515 examples [00:01, 27472.01 examples/s]
Generating train split: 52653 examples [00:02, 27445.93 examples/s]
Generating train s
1: plit: 56710 examples [00:02, 28666.50 examples/s]
Generating train split: 60909 examples [00:02, 29935.85 examples/s]
Generating train split: 65092 examples [00:02, 30803.71 examples/s]
Generating train split: 69142 examples [00:02, 31200.88 examples/s]
Generating train split: 73676 examples [00:02, 23831.52 examples/s]
Generating train split: 77664 examples [00:02, 24180.84 examples/s]
Generating train split: 81794 examples [00:03, 26147.90 examples/s]
Generating train split: 85981 examples [00:03, 27973.48 examples/s]
Generating train split: 90147 examples [00:03, 29296.76 examples/s]
Generating train split: 94388 examples [00:03, 30536.83 examples/s]
Generating train split: 98646 examples [00:03, 31447.66 examples/s]
Generating train split: 102828 examples [00:03, 32046.15 examples/s]
Generating train split: 106575 examples [00:03, 24353.82 examples/s]
Generating train split: 110735 examples [00:04, 24852.94 examples/s]
Generating train split: 114749 examples [00:04, 25589.16 examples/s]
Generating train s
1: plit: 118875 examples [00:04, 27308.12 examples/s]
Generating train split: 122971 examples [00:04, 28618.93 examples/s]
Generating train split: 127099 examples [00:04, 29706.63 examples/s]
Generating train split: 131236 examples [00:04, 30525.02 examples/s]
Generating train split: 135408 examples [00:04, 31254.57 examples/s]
Generating train split: 139320 examples [00:05, 31823.68 examples/s]
Generating train split: 143417 examples [00:05, 25573.03 examples/s]
Generating train split: 147621 examples [00:05, 25745.57 examples/s]
Generating train split: 151850 examples [00:05, 27497.88 examples/s]
Generating train split: 155963 examples [00:05, 27988.85 examples/s]
Generating train split: 160138 examples [00:05, 29336.05 examples/s]
Generating train split: 164196 examples [00:05, 30082.80 examples/s]
Generating train split: 168374 examples [00:06, 30850.17 examples/s]
Generating train split: 172513 examples [00:06, 31472.07 examples/s]
Generating train split: 176187 examples [00:06, 20717.29 examples/s]
Generat
1: ing train split: 180309 examples [00:06, 21679.74 examples/s]
Generating train split: 184479 examples [00:06, 23406.56 examples/s]
Generating train split: 188569 examples [00:06, 25565.05 examples/s]
Generating train split: 192644 examples [00:07, 27270.87 examples/s]
Generating train split: 196749 examples [00:07, 28812.56 examples/s]
Generating train split: 200936 examples [00:07, 29996.06 examples/s]
Generating train split: 205208 examples [00:07, 31151.71 examples/s]
Generating train split: 208979 examples [00:07, 30141.53 examples/s]
Generating train split: 213172 examples [00:07, 23110.96 examples/s]
Generating train split: 217397 examples [00:08, 23949.96 examples/s]
Generating train split: 221516 examples [00:08, 25897.33 examples/s]
Generating train split: 225610 examples [00:08, 27505.84 examples/s]
Generating train split: 229634 examples [00:08, 28592.83 examples/s]
Generating train split: 233748 examples [00:08, 29622.27 examples/s]
Generating train split: 238054 examples [00:08, 30758.02 examples
1: /s]
Generating train split: 242070 examples [00:08, 30991.11 examples/s]
Generating train split: 245907 examples [00:09, 20090.32 examples/s]
Generating train split: 250178 examples [00:09, 21802.30 examples/s]
Generating train split: 254319 examples [00:09, 23403.31 examples/s]
Generating train split: 257368 examples [00:09, 23472.95 examples/s]
Generating train split: 261480 examples [00:09, 25769.81 examples/s]
Generating train split: 265701 examples [00:09, 27742.77 examples/s]
Generating train split: 269895 examples [00:10, 29242.81 examples/s]
Generating train split: 274079 examples [00:10, 30389.73 examples/s]
Generating train split: 278193 examples [00:10, 31157.33 examples/s]
Generating train split: 282789 examples [00:10, 21226.15 examples/s]
Generating train split: 286994 examples [00:10, 22489.03 examples/s]
Generating train split: 291267 examples [00:10, 24935.14 examples/s]
Generating train split: 295386 examples [00:11, 26796.99 examples/s]
Generating train split: 299494 examples [00:11, 28272.
1: 85 examples/s]
Generating train split: 303712 examples [00:11, 29700.38 examples/s]
Generating train split: 307849 examples [00:11, 30597.22 examples/s]
Generating train split: 312045 examples [00:11, 31255.87 examples/s]
Generating train split: 315483 examples [00:11, 24728.17 examples/s]
Generating train split: 319662 examples [00:11, 25447.03 examples/s]
Generating train split: 323782 examples [00:12, 26220.27 examples/s]
Generating train split: 327874 examples [00:12, 27849.78 examples/s]
Generating train split: 331907 examples [00:12, 28967.50 examples/s]
Generating train split: 335997 examples [00:12, 29951.90 examples/s]
Generating train split: 340096 examples [00:12, 30810.03 examples/s]
Generating train split: 344306 examples [00:12, 31558.37 examples/s]
Generating train split: 348299 examples [00:12, 31998.04 examples/s]
Generating train split: 352404 examples [00:13, 25431.14 examples/s]
Generating train split: 356546 examples [00:13, 25589.66 examples/s]
Generating train split: 360733 examples [00
1: :13, 26674.43 examples/s]
Generating train split: 364723 examples [00:13, 28155.23 examples/s]
Generating train split: 368783 examples [00:13, 28490.40 examples/s]
Generating train split: 372952 examples [00:13, 29801.85 examples/s]
Generating train split: 377318 examples [00:13, 31180.27 examples/s]
Generating train split: 381500 examples [00:13, 31849.33 examples/s]
Generating train split: 385159 examples [00:14, 24399.03 examples/s]
Generating train split: 389294 examples [00:14, 25010.24 examples/s]
Generating train split: 393520 examples [00:14, 26173.95 examples/s]
Generating train split: 397645 examples [00:14, 27980.66 examples/s]
Generating train split: 401688 examples [00:14, 29154.64 examples/s]
Generating train split: 405875 examples [00:14, 30057.51 examples/s]
Generating train split: 410054 examples [00:15, 31075.21 examples/s]
Generating train split: 413273 examples [00:15, 30291.76 examples/s]
Generating train split: 417530 examples [00:15, 31363.50 examples/s]
Generating train split: 417958 e
1: xamples [00:15, 27355.07 examples/s]
1: [2025-11-23 14:41:47,913] [INFO] [axolotl.utils.data.wrappers.get_dataset_wrapper:88] [PID:2965584] [RANK:2] Loading dataset: /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/thinking with base_type: chat_template and prompt_style: None
1: 
Tokenizing Prompts (num_proc=32):   0%|          | 0/417958 [00:00<?, ? examples/s]
Tokenizing Prompts (num_proc=32):   0%|          | 1000/417958 [00:11<1:21:59, 84.76 examples/s]
Tokenizing Prompts (num_proc=32):   0%|          | 2000/417958 [00:11<34:15, 202.32 examples/s] 
Tokenizing Prompts (num_proc=32):   1%|          | 3000/417958 [00:12<19:32, 353.87 examples/s]
Tokenizing Prompts (num_proc=32):   1%|          | 4000/417958 [00:12<12:12, 564.89 examples/s]
Tokenizing Prompts (num_proc=32):   1%|          | 5000/417958 [00:12<08:31, 807.87 examples/s]
Tokenizing Prompts (num_proc=32):   1%|▏         | 6000/417958 [00:12<06:15, 1098.38 examples/s]
Tokenizing Prompts (num_proc=32):   2%|▏         | 7000/417958 [00:13<05:47, 1180.99 examples/s]
Tokenizing Prompts (num_proc=32):   2%|▏         | 9000/417958 [00:13<03:14, 2105.61 examples/s]
Tokenizing Prompts (num_proc=32):   2%|▏         | 10000/417958 [00:14<02:51, 2381.10 examples/s]
Tokenizing Prompts (num_proc=32):   3%|▎         | 12000/4
1: 17958 [00:14<01:49, 3719.10 examples/s]
Tokenizing Prompts (num_proc=32):   3%|▎         | 13000/417958 [00:15<02:56, 2289.70 examples/s]
Tokenizing Prompts (num_proc=32):   4%|▎         | 15000/417958 [00:15<01:57, 3416.71 examples/s]
Tokenizing Prompts (num_proc=32):   4%|▍         | 16000/417958 [00:15<01:45, 3808.89 examples/s]
Tokenizing Prompts (num_proc=32):   4%|▍         | 17000/417958 [00:15<02:01, 3304.83 examples/s]
Tokenizing Prompts (num_proc=32):   4%|▍         | 18000/417958 [00:16<02:20, 2849.54 examples/s]
Tokenizing Prompts (num_proc=32):   5%|▍         | 20000/417958 [00:16<01:48, 3673.63 examples/s]
Tokenizing Prompts (num_proc=32):   5%|▌         | 21000/417958 [00:17<01:56, 3417.52 examples/s]
Tokenizing Prompts (num_proc=32):   5%|▌         | 22000/417958 [00:17<02:29, 2655.73 examples/s]
Tokenizing Prompts (num_proc=32):   6%|▌         | 24000/417958 [00:18<02:05, 3150.01 examples/s]
Tokenizing Prompts (num_proc=32):   6%|▌         | 25000/417958 [00:18<01:52, 3502
1: .21 examples/s]
Tokenizing Prompts (num_proc=32):   6%|▌         | 26000/417958 [00:18<01:54, 3428.37 examples/s]
Tokenizing Prompts (num_proc=32):   6%|▋         | 27000/417958 [00:18<01:42, 3809.23 examples/s]
Tokenizing Prompts (num_proc=32):   7%|▋         | 28000/417958 [00:19<02:16, 2865.89 examples/s]
Tokenizing Prompts (num_proc=32):   7%|▋         | 29000/417958 [00:19<01:50, 3533.61 examples/s]
Tokenizing Prompts (num_proc=32):   7%|▋         | 30000/417958 [00:20<02:07, 3036.03 examples/s]
Tokenizing Prompts (num_proc=32):   8%|▊         | 32000/417958 [00:20<01:36, 4020.07 examples/s]
Tokenizing Prompts (num_proc=32):   8%|▊         | 33000/417958 [00:23<05:59, 1071.83 examples/s]
Tokenizing Prompts (num_proc=32):   8%|▊         | 34000/417958 [00:24<05:45, 1111.58 examples/s]
Tokenizing Prompts (num_proc=32):   9%|▊         | 36000/417958 [00:24<03:31, 1804.68 examples/s]
Tokenizing Prompts (num_proc=32):   9%|▉         | 37000/417958 [00:24<02:53, 2199.22 examples/s]
Tokenizi
1: ng Prompts (num_proc=32):   9%|▉         | 38000/417958 [00:24<02:55, 2160.44 examples/s]
Tokenizing Prompts (num_proc=32):   9%|▉         | 39000/417958 [00:25<02:25, 2595.92 examples/s]
Tokenizing Prompts (num_proc=32):  10%|▉         | 40000/417958 [00:25<02:45, 2286.81 examples/s]
Tokenizing Prompts (num_proc=32):  10%|▉         | 41000/417958 [00:25<02:10, 2881.03 examples/s]
Tokenizing Prompts (num_proc=32):  10%|█         | 42000/417958 [00:26<01:55, 3258.29 examples/s]
Tokenizing Prompts (num_proc=32):  11%|█         | 44000/417958 [00:26<01:41, 3669.37 examples/s]
Tokenizing Prompts (num_proc=32):  11%|█         | 46000/417958 [00:26<01:31, 4058.46 examples/s]
Tokenizing Prompts (num_proc=32):  11%|█▏        | 48000/417958 [00:27<01:17, 4782.50 examples/s]
Tokenizing Prompts (num_proc=32):  12%|█▏        | 49000/417958 [00:27<01:44, 3518.71 examples/s]
Tokenizing Prompts (num_proc=32):  12%|█▏        | 50000/417958 [00:28<02:00, 3046.11 examples/s]
Tokenizing Prompts (num_pr
1: oc=32):  12%|█▏        | 52000/417958 [00:28<01:45, 3461.77 examples/s]
Tokenizing Prompts (num_proc=32):  13%|█▎        | 54000/417958 [00:28<01:14, 4874.94 examples/s]
Tokenizing Prompts (num_proc=32):  13%|█▎        | 55000/417958 [00:28<01:07, 5366.61 examples/s]
Tokenizing Prompts (num_proc=32):  13%|█▎        | 56000/417958 [00:30<02:19, 2596.78 examples/s]
Tokenizing Prompts (num_proc=32):  14%|█▎        | 57000/417958 [00:30<03:10, 1898.90 examples/s]
Tokenizing Prompts (num_proc=32):  14%|█▍        | 59000/417958 [00:31<02:19, 2577.83 examples/s]
Tokenizing Prompts (num_proc=32):  14%|█▍        | 60000/417958 [00:31<01:56, 3075.41 examples/s]
Tokenizing Prompts (num_proc=32):  15%|█▍        | 61000/417958 [00:31<01:54, 3122.93 examples/s]
Tokenizing Prompts (num_proc=32):  15%|█▍        | 62000/417958 [00:32<01:46, 3355.92 examples/s]
Tokenizing Prompts (num_proc=32):  15%|█▌        | 63000/417958 [00:32<02:00, 2949.37 examples/s]
Tokenizing Prompts (num_proc=3
1: 2):  15%|█▌        | 64000/417958 [00:32<01:43, 3433.55 examples/s]
Tokenizing Prompts (num_proc=32):  16%|█▌        | 65000/417958 [00:35<06:28, 908.19 examples/s] 
Tokenizing Prompts (num_proc=32):  16%|█▌        | 66000/417958 [00:36<05:12, 1125.37 examples/s]
Tokenizing Prompts (num_proc=32):  16%|█▋        | 68000/417958 [00:36<03:04, 1899.67 examples/s]
Tokenizing Prompts (num_proc=32):  17%|█▋        | 69000/417958 [00:36<02:43, 2136.73 examples/s]
Tokenizing Prompts (num_proc=32):  17%|█▋        | 70000/417958 [00:36<02:35, 2244.16 examples/s]
Tokenizing Prompts (num_proc=32):  17%|█▋        | 71000/417958 [00:37<02:41, 2152.38 examples/s]
Tokenizing Prompts (num_proc=32):  17%|█▋        | 72000/417958 [00:37<02:18, 2492.13 examples/s]
Tokenizing Prompts (num_proc=32):  17%|█▋        | 73000/417958 [00:38<02:09, 2658.15 examples/s]
Tokenizing Prompts (num_proc=32):  18%|█▊        | 75000/417958 [00:38<01:20, 4242.42 examples/s]
Tokenizing Prompts (num_proc=32): 
1:  18%|█▊        | 77000/417958 [00:39<01:47, 3172.94 examples/s]
Tokenizing Prompts (num_proc=32):  19%|█▉        | 79000/417958 [00:39<01:18, 4303.54 examples/s]
Tokenizing Prompts (num_proc=32):  19%|█▉        | 80000/417958 [00:39<01:35, 3552.85 examples/s]
Tokenizing Prompts (num_proc=32):  19%|█▉        | 81000/417958 [00:40<01:50, 3043.62 examples/s]
Tokenizing Prompts (num_proc=32):  20%|█▉        | 83000/417958 [00:40<01:13, 4536.61 examples/s]
Tokenizing Prompts (num_proc=32):  20%|██        | 84000/417958 [00:40<01:32, 3612.02 examples/s]
Tokenizing Prompts (num_proc=32):  21%|██        | 87000/417958 [00:41<01:04, 5122.10 examples/s]
Tokenizing Prompts (num_proc=32):  21%|██        | 88000/417958 [00:42<02:01, 2726.54 examples/s]
Tokenizing Prompts (num_proc=32):  21%|██▏       | 89000/417958 [00:42<02:01, 2701.43 examples/s]
Tokenizing Prompts (num_proc=32):  22%|██▏       | 90000/417958 [00:43<02:05, 2615.76 examples/s]
Tokenizing Prompts (num_proc=32): 
1:  22%|██▏       | 91000/417958 [00:43<01:57, 2772.82 examples/s]
Tokenizing Prompts (num_proc=32):  22%|██▏       | 92000/417958 [00:43<01:59, 2732.17 examples/s]
Tokenizing Prompts (num_proc=32):  22%|██▏       | 93000/417958 [00:44<02:00, 2689.70 examples/s]
Tokenizing Prompts (num_proc=32):  23%|██▎       | 95000/417958 [00:44<01:44, 3097.01 examples/s]
Tokenizing Prompts (num_proc=32):  23%|██▎       | 96000/417958 [00:44<01:27, 3674.39 examples/s]
Tokenizing Prompts (num_proc=32):  23%|██▎       | 97000/417958 [00:47<05:06, 1047.53 examples/s]
Tokenizing Prompts (num_proc=32):  23%|██▎       | 98000/417958 [00:48<04:27, 1195.35 examples/s]
Tokenizing Prompts (num_proc=32):  24%|██▎       | 99000/417958 [00:48<03:53, 1365.99 examples/s]
Tokenizing Prompts (num_proc=32):  24%|██▍       | 101000/417958 [00:49<03:18, 1600.19 examples/s]
Tokenizing Prompts (num_proc=32):  24%|██▍       | 102000/417958 [00:49<02:44, 1923.33 examples/s]
Tokenizing Promp
1: ts (num_proc=32):  25%|██▍       | 104000/417958 [00:49<01:43, 3029.45 examples/s]
Tokenizing Prompts (num_proc=32):  25%|██▌       | 105000/417958 [00:50<01:39, 3143.37 examples/s]
Tokenizing Prompts (num_proc=32):  26%|██▌       | 108000/417958 [00:50<01:18, 3935.26 examples/s]
Tokenizing Prompts (num_proc=32):  26%|██▋       | 110000/417958 [00:50<01:02, 4923.64 examples/s]
Tokenizing Prompts (num_proc=32):  27%|██▋       | 111000/417958 [00:51<01:30, 3380.02 examples/s]
Tokenizing Prompts (num_proc=32):  27%|██▋       | 112000/417958 [00:51<01:23, 3644.16 examples/s]
Tokenizing Prompts (num_proc=32):  27%|██▋       | 113000/417958 [00:51<01:11, 4262.33 examples/s]
Tokenizing Prompts (num_proc=32):  27%|██▋       | 114000/417958 [00:52<01:08, 4428.68 examples/s]
Tokenizing Prompts (num_proc=32):  28%|██▊       | 115000/417958 [00:52<01:39, 3047.80 examples/s]
Tokenizing Prompts (num_proc=32):  28%|██▊       | 116000/417958 [00:52<01:25, 3546.21 ex
1: amples/s]
Tokenizing Prompts (num_proc=32):  28%|██▊       | 117000/417958 [00:53<01:30, 3340.34 examples/s]
Tokenizing Prompts (num_proc=32):  28%|██▊       | 119000/417958 [00:53<01:07, 4432.83 examples/s]
Tokenizing Prompts (num_proc=32):  29%|██▊       | 120000/417958 [00:54<01:43, 2873.49 examples/s]
Tokenizing Prompts (num_proc=32):  29%|██▉       | 121000/417958 [00:54<01:48, 2737.85 examples/s]
Tokenizing Prompts (num_proc=32):  29%|██▉       | 122000/417958 [00:55<02:05, 2350.15 examples/s]
Tokenizing Prompts (num_proc=32):  30%|██▉       | 124000/417958 [00:55<01:23, 3519.35 examples/s]
Tokenizing Prompts (num_proc=32):  30%|██▉       | 125000/417958 [00:55<01:20, 3621.69 examples/s]
Tokenizing Prompts (num_proc=32):  30%|███       | 126000/417958 [00:56<02:01, 2408.91 examples/s]
Tokenizing Prompts (num_proc=32):  30%|███       | 127000/417958 [00:57<02:19, 2093.02 examples/s]
Tokenizing Prompts (num_proc=32):  31%|███       | 129000/41795
1: 8 [00:58<03:02, 1579.53 examples/s]
Tokenizing Prompts (num_proc=32):  31%|███       | 130000/417958 [00:59<03:29, 1371.78 examples/s]
Tokenizing Prompts (num_proc=32):  31%|███▏      | 131000/417958 [01:00<02:55, 1638.46 examples/s]
Tokenizing Prompts (num_proc=32):  32%|███▏      | 132000/417958 [01:01<03:26, 1388.10 examples/s]
Tokenizing Prompts (num_proc=32):  32%|███▏      | 133000/417958 [01:01<02:41, 1764.83 examples/s]
Tokenizing Prompts (num_proc=32):  32%|███▏      | 134000/417958 [01:01<02:40, 1766.05 examples/s]
Tokenizing Prompts (num_proc=32):  32%|███▏      | 135000/417958 [01:02<02:06, 2244.21 examples/s]
Tokenizing Prompts (num_proc=32):  33%|███▎      | 136000/417958 [01:02<01:57, 2404.98 examples/s]
Tokenizing Prompts (num_proc=32):  33%|███▎      | 137000/417958 [01:02<01:40, 2788.06 examples/s]
Tokenizing Prompts (num_proc=32):  33%|███▎      | 138000/417958 [01:02<01:34, 2965.18 examples/s]
Tokenizing Prompts (num_pro
1: c=32):  33%|███▎      | 140000/417958 [01:03<01:11, 3878.88 examples/s]
Tokenizing Prompts (num_proc=32):  34%|███▍      | 142000/417958 [01:03<00:49, 5522.30 examples/s]
Tokenizing Prompts (num_proc=32):  34%|███▍      | 143000/417958 [01:03<00:46, 5897.89 examples/s]
Tokenizing Prompts (num_proc=32):  35%|███▍      | 145000/417958 [01:03<00:51, 5332.91 examples/s]
Tokenizing Prompts (num_proc=32):  35%|███▍      | 146000/417958 [01:04<01:10, 3881.24 examples/s]
Tokenizing Prompts (num_proc=32):  35%|███▌      | 147000/417958 [01:04<01:13, 3691.96 examples/s]
Tokenizing Prompts (num_proc=32):  35%|███▌      | 148000/417958 [01:05<01:20, 3341.06 examples/s]
Tokenizing Prompts (num_proc=32):  36%|███▌      | 149000/417958 [01:05<01:17, 3484.67 examples/s]
Tokenizing Prompts (num_proc=32):  36%|███▌      | 150000/417958 [01:05<01:18, 3429.35 examples/s]
Tokenizing Prompts (num_proc=32):  36%|███▌      | 151000/417958 [01:06<01:21, 3
1: 261.91 examples/s]
Tokenizing Prompts (num_proc=32):  36%|███▋      | 152000/417958 [01:06<01:14, 3547.28 examples/s]
Tokenizing Prompts (num_proc=32):  37%|███▋      | 153000/417958 [01:06<01:44, 2532.94 examples/s]
Tokenizing Prompts (num_proc=32):  37%|███▋      | 154000/417958 [01:07<01:59, 2217.42 examples/s]
Tokenizing Prompts (num_proc=32):  37%|███▋      | 155000/417958 [01:07<01:57, 2245.02 examples/s]
Tokenizing Prompts (num_proc=32):  37%|███▋      | 156000/417958 [01:08<02:11, 1994.27 examples/s]
Tokenizing Prompts (num_proc=32):  38%|███▊      | 158000/417958 [01:08<01:25, 3033.20 examples/s]
Tokenizing Prompts (num_proc=32):  38%|███▊      | 159000/417958 [01:09<01:59, 2171.48 examples/s]
Tokenizing Prompts (num_proc=32):  39%|███▊      | 161000/417958 [01:09<01:15, 3411.03 examples/s]
Tokenizing Prompts (num_proc=32):  39%|███▉      | 162000/417958 [01:10<02:00, 2125.66 examples/s]
Tokenizing Prompts (num_proc=32):  39%|█
1: ██▉      | 163000/417958 [01:13<03:53, 1090.43 examples/s]
Tokenizing Prompts (num_proc=32):  39%|███▉      | 165000/417958 [01:13<02:39, 1589.91 examples/s]
Tokenizing Prompts (num_proc=32):  40%|███▉      | 167000/417958 [01:13<01:46, 2360.86 examples/s]
Tokenizing Prompts (num_proc=32):  40%|████      | 168000/417958 [01:14<02:10, 1918.64 examples/s]
Tokenizing Prompts (num_proc=32):  41%|████      | 170000/417958 [01:14<01:34, 2634.96 examples/s]
Tokenizing Prompts (num_proc=32):  41%|████▏     | 173000/417958 [01:15<01:00, 4026.51 examples/s]
Tokenizing Prompts (num_proc=32):  42%|████▏     | 175000/417958 [01:15<00:59, 4092.04 examples/s]
Tokenizing Prompts (num_proc=32):  43%|████▎     | 178000/417958 [01:16<00:48, 4907.82 examples/s]
Tokenizing Prompts (num_proc=32):  43%|████▎     | 179000/417958 [01:16<01:10, 3394.15 examples/s]
Tokenizing Prompts (num_proc=32):  43%|████▎     | 180000/417958 [01:17<01:17, 3090.1
1: 6 examples/s]
Tokenizing Prompts (num_proc=32):  43%|████▎     | 181000/417958 [01:17<01:16, 3090.84 examples/s]
Tokenizing Prompts (num_proc=32):  44%|████▍     | 184000/417958 [01:18<01:18, 2978.06 examples/s]
Tokenizing Prompts (num_proc=32):  44%|████▍     | 185000/417958 [01:18<01:17, 3001.21 examples/s]
Tokenizing Prompts (num_proc=32):  45%|████▍     | 186000/417958 [01:20<01:57, 1978.24 examples/s]
Tokenizing Prompts (num_proc=32):  45%|████▍     | 188000/417958 [01:20<01:19, 2885.81 examples/s]
Tokenizing Prompts (num_proc=32):  45%|████▌     | 189000/417958 [01:20<01:20, 2833.15 examples/s]
Tokenizing Prompts (num_proc=32):  45%|████▌     | 190000/417958 [01:20<01:09, 3279.45 examples/s]
Tokenizing Prompts (num_proc=32):  46%|████▌     | 191000/417958 [01:21<01:08, 3317.55 examples/s]
Tokenizing Prompts (num_proc=32):  46%|████▌     | 192000/417958 [01:21<00:59, 3767.24 examples/s]
Tokenizing Prompts (num_proc=
1: 32):  46%|████▌     | 193000/417958 [01:21<01:14, 3032.44 examples/s]
Tokenizing Prompts (num_proc=32):  46%|████▋     | 194000/417958 [01:22<01:56, 1917.15 examples/s]
Tokenizing Prompts (num_proc=32):  47%|████▋     | 195000/417958 [01:25<03:46, 986.07 examples/s] 
Tokenizing Prompts (num_proc=32):  47%|████▋     | 196000/417958 [01:25<02:55, 1262.47 examples/s]
Tokenizing Prompts (num_proc=32):  47%|████▋     | 198000/417958 [01:25<01:50, 1988.48 examples/s]
Tokenizing Prompts (num_proc=32):  48%|████▊     | 199000/417958 [01:25<01:42, 2137.62 examples/s]
Tokenizing Prompts (num_proc=32):  48%|████▊     | 200000/417958 [01:26<01:30, 2402.33 examples/s]
Tokenizing Prompts (num_proc=32):  48%|████▊     | 201000/417958 [01:27<01:53, 1918.60 examples/s]
Tokenizing Prompts (num_proc=32):  48%|████▊     | 202000/417958 [01:27<01:39, 2173.26 examples/s]
Tokenizing Prompts (num_proc=32):  49%|████▊     | 203000/4179
1: 58 [01:27<01:21, 2633.30 examples/s]
Tokenizing Prompts (num_proc=32):  49%|████▉     | 206000/417958 [01:27<00:43, 4845.33 examples/s]
Tokenizing Prompts (num_proc=32):  50%|████▉     | 207000/417958 [01:27<00:44, 4748.16 examples/s]
Tokenizing Prompts (num_proc=32):  50%|████▉     | 208000/417958 [01:28<00:53, 3932.99 examples/s]
Tokenizing Prompts (num_proc=32):  50%|█████     | 209000/417958 [01:28<00:53, 3898.93 examples/s]
Tokenizing Prompts (num_proc=32):  50%|█████     | 210000/417958 [01:28<00:48, 4245.00 examples/s]
Tokenizing Prompts (num_proc=32):  50%|█████     | 211000/417958 [01:29<01:04, 3232.92 examples/s]
Tokenizing Prompts (num_proc=32):  51%|█████     | 213000/417958 [01:29<01:06, 3102.83 examples/s]
Tokenizing Prompts (num_proc=32):  51%|█████▏    | 215000/417958 [01:30<01:09, 2911.61 examples/s]
Tokenizing Prompts (num_proc=32):  52%|█████▏    | 216000/417958 [01:31<01:19, 2528.70 examples/s]
To
1: kenizing Prompts (num_proc=32):  52%|█████▏    | 218000/417958 [01:32<01:20, 2485.53 examples/s]
Tokenizing Prompts (num_proc=32):  52%|█████▏    | 219000/417958 [01:33<01:43, 1925.07 examples/s]
Tokenizing Prompts (num_proc=32):  53%|█████▎    | 222000/417958 [01:33<00:57, 3436.26 examples/s]
Tokenizing Prompts (num_proc=32):  54%|█████▎    | 224000/417958 [01:33<00:51, 3759.15 examples/s]
Tokenizing Prompts (num_proc=32):  54%|█████▍    | 225000/417958 [01:34<01:03, 3059.79 examples/s]
Tokenizing Prompts (num_proc=32):  54%|█████▍    | 226000/417958 [01:34<01:17, 2482.12 examples/s]
Tokenizing Prompts (num_proc=32):  54%|█████▍    | 227000/417958 [01:37<02:33, 1240.33 examples/s]
Tokenizing Prompts (num_proc=32):  55%|█████▍    | 228000/417958 [01:37<02:25, 1301.20 examples/s]
Tokenizing Prompts (num_proc=32):  55%|█████▍    | 229000/417958 [01:37<01:53, 1663.53 examples/s]
Tokenizing Prompts (num_pro
1: c=32):  55%|█████▌    | 231000/417958 [01:38<01:10, 2664.63 examples/s]
Tokenizing Prompts (num_proc=32):  56%|█████▌    | 232000/417958 [01:38<01:35, 1955.50 examples/s]
Tokenizing Prompts (num_proc=32):  56%|█████▌    | 233000/417958 [01:39<01:38, 1871.55 examples/s]
Tokenizing Prompts (num_proc=32):  56%|█████▌    | 234000/417958 [01:39<01:23, 2198.21 examples/s]
Tokenizing Prompts (num_proc=32):  56%|█████▋    | 236000/417958 [01:40<01:05, 2779.35 examples/s]
Tokenizing Prompts (num_proc=32):  57%|█████▋    | 238000/417958 [01:40<00:56, 3193.33 examples/s]
Tokenizing Prompts (num_proc=32):  57%|█████▋    | 239000/417958 [01:40<00:50, 3557.15 examples/s]
Tokenizing Prompts (num_proc=32):  57%|█████▋    | 240000/417958 [01:41<00:43, 4136.61 examples/s]
Tokenizing Prompts (num_proc=32):  58%|█████▊    | 241000/417958 [01:41<00:38, 4635.28 examples/s]
Tokenizing Prompts (num_proc=32):  58%|████�
1: ��▊    | 242000/417958 [01:41<00:36, 4858.40 examples/s]
Tokenizing Prompts (num_proc=32):  58%|█████▊    | 244000/417958 [01:41<00:28, 6149.92 examples/s]
Tokenizing Prompts (num_proc=32):  59%|█████▊    | 245000/417958 [01:42<00:48, 3535.94 examples/s]
Tokenizing Prompts (num_proc=32):  59%|█████▉    | 246000/417958 [01:42<01:02, 2757.12 examples/s]
Tokenizing Prompts (num_proc=32):  59%|█████▉    | 248000/417958 [01:43<01:06, 2538.41 examples/s]
Tokenizing Prompts (num_proc=32):  60%|█████▉    | 249000/417958 [01:43<00:58, 2907.23 examples/s]
Tokenizing Prompts (num_proc=32):  60%|█████▉    | 250000/417958 [01:44<01:02, 2687.54 examples/s]
Tokenizing Prompts (num_proc=32):  60%|██████    | 251000/417958 [01:45<01:42, 1636.05 examples/s]
Tokenizing Prompts (num_proc=32):  61%|██████    | 253000/417958 [01:45<01:04, 2574.89 examples/s]
Tokenizing Prompts (num_proc=32):  61%|██████    | 255000/417958 
1: [01:45<00:42, 3816.80 examples/s]
Tokenizing Prompts (num_proc=32):  61%|██████▏   | 256000/417958 [01:46<00:47, 3399.91 examples/s]
Tokenizing Prompts (num_proc=32):  61%|██████▏   | 257000/417958 [01:46<00:40, 4009.49 examples/s]
Tokenizing Prompts (num_proc=32):  62%|██████▏   | 258000/417958 [01:46<00:40, 3938.25 examples/s]
Tokenizing Prompts (num_proc=32):  62%|██████▏   | 259000/417958 [01:47<01:20, 1980.69 examples/s]
Tokenizing Prompts (num_proc=32):  62%|██████▏   | 260000/417958 [01:49<01:51, 1416.83 examples/s]
Tokenizing Prompts (num_proc=32):  62%|██████▏   | 261000/417958 [01:49<01:44, 1509.05 examples/s]
Tokenizing Prompts (num_proc=32):  63%|██████▎   | 262000/417958 [01:50<01:28, 1752.67 examples/s]
Tokenizing Prompts (num_proc=32):  63%|██████▎   | 263000/417958 [01:50<01:16, 2018.53 examples/s]
Tokenizing Prompts (num_proc=32):  63%|██████▎   | 264000/417958 [01:50<
1: 01:22, 1859.66 examples/s]
Tokenizing Prompts (num_proc=32):  63%|██████▎   | 265000/417958 [01:51<01:29, 1700.45 examples/s]
Tokenizing Prompts (num_proc=32):  64%|██████▎   | 266000/417958 [01:51<01:07, 2251.10 examples/s]
Tokenizing Prompts (num_proc=32):  64%|██████▍   | 267000/417958 [01:52<00:57, 2603.29 examples/s]
Tokenizing Prompts (num_proc=32):  64%|██████▍   | 268000/417958 [01:52<00:51, 2905.36 examples/s]
Tokenizing Prompts (num_proc=32):  64%|██████▍   | 269000/417958 [01:52<00:59, 2521.70 examples/s]
Tokenizing Prompts (num_proc=32):  65%|██████▍   | 271000/417958 [01:52<00:35, 4113.41 examples/s]
Tokenizing Prompts (num_proc=32):  65%|██████▌   | 272000/417958 [01:53<00:42, 3462.00 examples/s]
Tokenizing Prompts (num_proc=32):  65%|██████▌   | 273000/417958 [01:53<00:54, 2636.36 examples/s]
Tokenizing Prompts (num_proc=32):  66%|██████▌   | 276000/417958 [01:54<00:35, 
1: 3996.23 examples/s]
Tokenizing Prompts (num_proc=32):  67%|██████▋   | 278000/417958 [01:55<00:50, 2798.09 examples/s]
Tokenizing Prompts (num_proc=32):  67%|██████▋   | 279000/417958 [01:55<00:43, 3180.33 examples/s]
Tokenizing Prompts (num_proc=32):  67%|██████▋   | 281000/417958 [01:56<00:37, 3700.30 examples/s]
Tokenizing Prompts (num_proc=32):  67%|██████▋   | 282000/417958 [01:57<01:04, 2106.95 examples/s]
Tokenizing Prompts (num_proc=32):  68%|██████▊   | 283000/417958 [01:57<00:58, 2322.52 examples/s]
Tokenizing Prompts (num_proc=32):  68%|██████▊   | 284000/417958 [01:58<00:58, 2297.90 examples/s]
Tokenizing Prompts (num_proc=32):  68%|██████▊   | 285000/417958 [01:58<00:47, 2819.23 examples/s]
Tokenizing Prompts (num_proc=32):  69%|██████▊   | 287000/417958 [01:58<00:30, 4334.31 examples/s]
Tokenizing Prompts (num_proc=32):  69%|██████▉   | 289000/417958 [01:58<00:24, 5354.38
1:  examples/s]
Tokenizing Prompts (num_proc=32):  69%|██████▉   | 290000/417958 [01:58<00:24, 5168.45 examples/s]
Tokenizing Prompts (num_proc=32):  70%|██████▉   | 291000/417958 [01:59<00:33, 3768.21 examples/s]
Tokenizing Prompts (num_proc=32):  70%|██████▉   | 292000/417958 [02:01<01:21, 1546.00 examples/s]
Tokenizing Prompts (num_proc=32):  70%|███████   | 293000/417958 [02:02<01:46, 1176.65 examples/s]
Tokenizing Prompts (num_proc=32):  71%|███████   | 295000/417958 [02:02<01:10, 1746.83 examples/s]
Tokenizing Prompts (num_proc=32):  71%|███████   | 297000/417958 [02:03<00:55, 2193.97 examples/s]
Tokenizing Prompts (num_proc=32):  71%|███████▏  | 298000/417958 [02:04<00:58, 2067.41 examples/s]
Tokenizing Prompts (num_proc=32):  72%|███████▏  | 299000/417958 [02:04<00:56, 2097.09 examples/s]
Tokenizing Prompts (num_proc=32):  72%|███████▏  | 301000/417958 [02:04<00:37, 3135.84 
1: examples/s]
Tokenizing Prompts (num_proc=32):  72%|███████▏  | 302000/417958 [02:04<00:37, 3112.19 examples/s]
Tokenizing Prompts (num_proc=32):  72%|███████▏  | 303000/417958 [02:05<00:36, 3109.14 examples/s]
Tokenizing Prompts (num_proc=32):  73%|███████▎  | 305000/417958 [02:05<00:24, 4664.26 examples/s]
Tokenizing Prompts (num_proc=32):  73%|███████▎  | 307000/417958 [02:06<00:42, 2609.48 examples/s]
Tokenizing Prompts (num_proc=32):  74%|███████▎  | 308000/417958 [02:07<00:42, 2567.67 examples/s]
Tokenizing Prompts (num_proc=32):  74%|███████▍  | 310000/417958 [02:07<00:40, 2664.75 examples/s]
Tokenizing Prompts (num_proc=32):  74%|███████▍  | 311000/417958 [02:08<00:35, 3038.61 examples/s]
Tokenizing Prompts (num_proc=32):  75%|███████▍  | 312000/417958 [02:08<00:31, 3335.50 examples/s]
Tokenizing Prompts (num_proc=32):  75%|███████▍  | 313000/417958 [02:08<00:2
1: 7, 3825.95 examples/s]
Tokenizing Prompts (num_proc=32):  75%|███████▌  | 314000/417958 [02:09<00:45, 2267.27 examples/s]
Tokenizing Prompts (num_proc=32):  75%|███████▌  | 315000/417958 [02:09<00:49, 2087.28 examples/s]
Tokenizing Prompts (num_proc=32):  76%|███████▌  | 316000/417958 [02:10<00:39, 2579.00 examples/s]
Tokenizing Prompts (num_proc=32):  76%|███████▌  | 318000/417958 [02:10<00:25, 3890.93 examples/s]
Tokenizing Prompts (num_proc=32):  76%|███████▋  | 319000/417958 [02:10<00:23, 4195.80 examples/s]
Tokenizing Prompts (num_proc=32):  77%|███████▋  | 320000/417958 [02:10<00:19, 4910.68 examples/s]
Tokenizing Prompts (num_proc=32):  77%|███████▋  | 321000/417958 [02:11<00:29, 3326.19 examples/s]
Tokenizing Prompts (num_proc=32):  77%|███████▋  | 322000/417958 [02:11<00:23, 4023.53 examples/s]
Tokenizing Prompts (num_proc=32):  77%|███████▋  | 323000/417958 
1: [02:11<00:20, 4596.64 examples/s]
Tokenizing Prompts (num_proc=32):  78%|███████▊  | 324000/417958 [02:13<01:02, 1503.59 examples/s]
Tokenizing Prompts (num_proc=32):  78%|███████▊  | 325000/417958 [02:13<01:03, 1462.23 examples/s]
Tokenizing Prompts (num_proc=32):  78%|███████▊  | 326000/417958 [02:14<01:05, 1413.38 examples/s]
Tokenizing Prompts (num_proc=32):  78%|███████▊  | 327000/417958 [02:14<00:54, 1681.73 examples/s]
Tokenizing Prompts (num_proc=32):  78%|███████▊  | 328000/417958 [02:15<00:54, 1648.85 examples/s]
Tokenizing Prompts (num_proc=32):  79%|███████▊  | 329000/417958 [02:16<00:57, 1549.39 examples/s]
Tokenizing Prompts (num_proc=32):  79%|███████▉  | 330000/417958 [02:16<00:43, 2010.40 examples/s]
Tokenizing Prompts (num_proc=32):  79%|███████▉  | 331000/417958 [02:16<00:34, 2541.25 examples/s]
Tokenizing Prompts (num_proc=32):  79%|███████▉  | 332
1: 000/417958 [02:16<00:26, 3219.02 examples/s]
Tokenizing Prompts (num_proc=32):  80%|████████  | 335000/417958 [02:17<00:22, 3629.68 examples/s]
Tokenizing Prompts (num_proc=32):  81%|████████  | 337000/417958 [02:18<00:27, 2910.10 examples/s]
Tokenizing Prompts (num_proc=32):  81%|████████  | 339000/417958 [02:18<00:19, 4009.91 examples/s]
Tokenizing Prompts (num_proc=32):  81%|████████▏ | 340000/417958 [02:19<00:27, 2878.47 examples/s]
Tokenizing Prompts (num_proc=32):  82%|████████▏ | 341000/417958 [02:19<00:24, 3113.59 examples/s]
Tokenizing Prompts (num_proc=32):  82%|████████▏ | 342000/417958 [02:20<00:29, 2555.88 examples/s]
Tokenizing Prompts (num_proc=32):  82%|████████▏ | 344000/417958 [02:20<00:25, 2870.21 examples/s]
Tokenizing Prompts (num_proc=32):  83%|████████▎ | 345000/417958 [02:20<00:22, 3199.86 examples/s]
Tokenizing Prompts (num_proc=32):  83%|███�
1: ��████▎ | 346000/417958 [02:21<00:20, 3579.31 examples/s]
Tokenizing Prompts (num_proc=32):  83%|████████▎ | 347000/417958 [02:21<00:19, 3676.09 examples/s]
Tokenizing Prompts (num_proc=32):  83%|████████▎ | 348000/417958 [02:21<00:21, 3199.36 examples/s]
Tokenizing Prompts (num_proc=32):  84%|████████▎ | 349000/417958 [02:21<00:19, 3526.14 examples/s]
Tokenizing Prompts (num_proc=32):  84%|████████▎ | 350000/417958 [02:22<00:34, 1987.21 examples/s]
Tokenizing Prompts (num_proc=32):  84%|████████▍ | 351000/417958 [02:23<00:29, 2299.92 examples/s]
Tokenizing Prompts (num_proc=32):  84%|████████▍ | 352000/417958 [02:23<00:27, 2378.30 examples/s]
Tokenizing Prompts (num_proc=32):  85%|████████▍ | 355000/417958 [02:24<00:19, 3234.12 examples/s]
Tokenizing Prompts (num_proc=32):  85%|████████▌ | 356000/417958 [02:24<00:21, 2913.53 examples/s]
Tokenizing Prompts (
1: num_proc=32):  85%|████████▌ | 357000/417958 [02:25<00:21, 2786.08 examples/s]
Tokenizing Prompts (num_proc=32):  86%|████████▌ | 358000/417958 [02:27<00:46, 1298.39 examples/s]
Tokenizing Prompts (num_proc=32):  86%|████████▌ | 360000/417958 [02:27<00:27, 2074.67 examples/s]
Tokenizing Prompts (num_proc=32):  86%|████████▋ | 361000/417958 [02:27<00:26, 2117.38 examples/s]
Tokenizing Prompts (num_proc=32):  87%|████████▋ | 362000/417958 [02:28<00:23, 2416.06 examples/s]
Tokenizing Prompts (num_proc=32):  87%|████████▋ | 363061/417958 [02:28<00:27, 2009.86 examples/s]
Tokenizing Prompts (num_proc=32):  88%|████████▊ | 366061/417958 [02:28<00:14, 3662.77 examples/s]
Tokenizing Prompts (num_proc=32):  88%|████████▊ | 367061/417958 [02:29<00:15, 3346.35 examples/s]
Tokenizing Prompts (num_proc=32):  88%|████████▊ | 368061/417958 [02:29<00:16, 3024.74 exa
1: mples/s]
Tokenizing Prompts (num_proc=32):  88%|████████▊ | 369061/417958 [02:30<00:18, 2594.06 examples/s]
Tokenizing Prompts (num_proc=32):  89%|████████▊ | 370061/417958 [02:31<00:22, 2146.36 examples/s]
Tokenizing Prompts (num_proc=32):  89%|████████▉ | 371061/417958 [02:31<00:21, 2232.68 examples/s]
Tokenizing Prompts (num_proc=32):  89%|████████▉ | 372061/417958 [02:31<00:20, 2236.76 examples/s]
Tokenizing Prompts (num_proc=32):  89%|████████▉ | 373061/417958 [02:32<00:20, 2223.84 examples/s]
Tokenizing Prompts (num_proc=32):  89%|████████▉ | 374061/417958 [02:32<00:16, 2635.62 examples/s]
Tokenizing Prompts (num_proc=32):  90%|████████▉ | 375061/417958 [02:32<00:13, 3222.41 examples/s]
Tokenizing Prompts (num_proc=32):  90%|████████▉ | 376061/417958 [02:33<00:12, 3336.52 examples/s]
Tokenizing Prompts (num_proc=32):  90%|█████████ | 378122/417
1: 958 [02:33<00:08, 4603.93 examples/s]
Tokenizing Prompts (num_proc=32):  91%|█████████ | 379184/417958 [02:33<00:07, 5289.04 examples/s]
Tokenizing Prompts (num_proc=32):  91%|█████████ | 380184/417958 [02:34<00:11, 3351.10 examples/s]
Tokenizing Prompts (num_proc=32):  91%|█████████▏| 382184/417958 [02:35<00:13, 2590.95 examples/s]
Tokenizing Prompts (num_proc=32):  92%|█████████▏| 383184/417958 [02:35<00:14, 2465.81 examples/s]
Tokenizing Prompts (num_proc=32):  92%|█████████▏| 384184/417958 [02:35<00:12, 2665.58 examples/s]
Tokenizing Prompts (num_proc=32):  92%|█████████▏| 385184/417958 [02:35<00:10, 3154.18 examples/s]
Tokenizing Prompts (num_proc=32):  92%|█████████▏| 386184/417958 [02:36<00:12, 2540.84 examples/s]
Tokenizing Prompts (num_proc=32):  93%|█████████▎| 387184/417958 [02:36<00:09, 3104.69 examples/s]
Tokenizing Prompts (num_proc=32):  93%
1: |█████████▎| 389245/417958 [02:37<00:09, 2907.10 examples/s]
Tokenizing Prompts (num_proc=32):  93%|█████████▎| 390306/417958 [02:38<00:17, 1618.54 examples/s]
Tokenizing Prompts (num_proc=32):  94%|█████████▎| 391306/417958 [02:39<00:16, 1626.09 examples/s]
Tokenizing Prompts (num_proc=32):  94%|█████████▍| 392368/417958 [02:39<00:13, 1948.96 examples/s]
Tokenizing Prompts (num_proc=32):  94%|█████████▍| 394368/417958 [02:39<00:07, 3108.44 examples/s]
Tokenizing Prompts (num_proc=32):  95%|█████████▍| 395430/417958 [02:40<00:08, 2787.47 examples/s]
Tokenizing Prompts (num_proc=32):  95%|█████████▍| 396675/417958 [02:41<00:09, 2175.01 examples/s]
Tokenizing Prompts (num_proc=32):  95%|█████████▌| 397736/417958 [02:42<00:10, 1877.26 examples/s]
Tokenizing Prompts (num_proc=32):  95%|█████████▌| 398797/417958 [02:42<00:08, 2383.13 exa
1: mples/s]
Tokenizing Prompts (num_proc=32):  96%|█████████▌| 400797/417958 [02:42<00:05, 3398.64 examples/s]
Tokenizing Prompts (num_proc=32):  96%|█████████▌| 401981/417958 [02:43<00:05, 2691.93 examples/s]
Tokenizing Prompts (num_proc=32):  96%|█████████▋| 402981/417958 [02:43<00:05, 2686.38 examples/s]
Tokenizing Prompts (num_proc=32):  97%|█████████▋| 404981/417958 [02:43<00:03, 3306.96 examples/s]
Tokenizing Prompts (num_proc=32):  97%|█████████▋| 406226/417958 [02:45<00:05, 2206.13 examples/s]
Tokenizing Prompts (num_proc=32):  97%|█████████▋| 407226/417958 [02:45<00:04, 2392.45 examples/s]
Tokenizing Prompts (num_proc=32):  98%|█████████▊| 409226/417958 [02:45<00:02, 3390.49 examples/s]
Tokenizing Prompts (num_proc=32):  98%|█████████▊| 410226/417958 [02:45<00:01, 3899.35 examples/s]
Tokenizing Prompts (num_proc=32):  98%|████████
1: █▊| 411409/417958 [02:46<00:02, 3056.94 examples/s]
Tokenizing Prompts (num_proc=32):  99%|█████████▊| 412531/417958 [02:46<00:01, 3585.92 examples/s]
Tokenizing Prompts (num_proc=32):  99%|█████████▉| 413653/417958 [02:47<00:02, 2141.83 examples/s]
Tokenizing Prompts (num_proc=32):  99%|█████████▉| 415653/417958 [02:47<00:00, 2892.17 examples/s]
Tokenizing Prompts (num_proc=32): 100%|█████████▉| 416653/417958 [02:48<00:00, 3093.16 examples/s]
Tokenizing Prompts (num_proc=32): 100%|█████████▉| 417714/417958 [02:48<00:00, 3734.98 examples/s]
Tokenizing Prompts (num_proc=32): 100%|██████████| 417958/417958 [02:49<00:00, 2471.37 examples/s]
1: 
Dropping Long Sequences (>16384) (num_proc=32):   0%|          | 0/557278 [00:00<?, ? examples/s]
Dropping Long Sequences (>16384) (num_proc=32):   0%|          | 1000/557278 [00:00<05:12, 1782.78 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):   1%|▏         | 7000/557278 [00:00<00:41, 13345.77 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):   2%|▏         | 11000/557278 [00:00<00:28, 18855.90 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):   3%|▎         | 17000/557278 [00:00<00:22, 24397.07 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):   4%|▍         | 24000/557278 [00:01<00:15, 33709.33 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):   5%|▌         | 29000/557278 [00:01<00:16, 32457.57 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):   6%|▌         | 33000/557278 [00:01<00:15, 33976.13 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):   7%|▋         | 40000/557278 [00:01<00:12, 42064.90 examples/s]
Dropping L
1: ong Sequences (>16384) (num_proc=32):   8%|▊         | 45000/557278 [00:01<00:11, 43688.28 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  10%|▉         | 54000/557278 [00:01<00:09, 54270.51 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  11%|█         | 60000/557278 [00:01<00:09, 54278.58 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  12%|█▏        | 68000/557278 [00:01<00:08, 57011.78 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  13%|█▎        | 75000/557278 [00:02<00:08, 59258.33 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  15%|█▍        | 83000/557278 [00:02<00:07, 63350.30 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  17%|█▋        | 92000/557278 [00:02<00:06, 67088.13 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  18%|█▊        | 100000/557278 [00:02<00:06, 66738.94 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  20%|█▉        | 109000/557278 [00:02<00:06, 67269.
1: 55 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  21%|██        | 117000/557278 [00:02<00:06, 68678.72 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  23%|██▎       | 126000/557278 [00:02<00:06, 70476.67 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  24%|██▍       | 134000/557278 [00:02<00:06, 63608.15 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  25%|██▌       | 142000/557278 [00:02<00:06, 64686.54 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  27%|██▋       | 150000/557278 [00:03<00:06, 67365.46 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  29%|██▊       | 159000/557278 [00:03<00:05, 70549.51 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  30%|███       | 168000/557278 [00:03<00:05, 74078.25 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  32%|███▏      | 176000/557278 [00:03<00:05, 67639.28 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  
1: 33%|███▎      | 184000/557278 [00:03<00:05, 68570.61 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  34%|███▍      | 191000/557278 [00:03<00:05, 67207.60 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  36%|███▌      | 200660/557278 [00:03<00:04, 74822.67 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  37%|███▋      | 208320/557278 [00:03<00:05, 58276.52 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  39%|███▊      | 215320/557278 [00:04<00:06, 54115.14 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  40%|███▉      | 221320/557278 [00:04<00:08, 37925.35 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  41%|████      | 226320/557278 [00:04<00:10, 33017.33 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  41%|████▏     | 230320/557278 [00:04<00:10, 30278.29 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  42%|████▏     | 235320/557278 [00:04
1: <00:09, 32772.62 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  43%|████▎     | 240320/557278 [00:05<00:09, 33964.76 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  44%|████▍     | 244320/557278 [00:05<00:11, 28406.81 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  45%|████▍     | 248320/557278 [00:05<00:10, 28867.38 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  45%|████▌     | 252320/557278 [00:05<00:10, 28331.40 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  46%|████▌     | 255320/557278 [00:05<00:10, 27478.74 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  47%|████▋     | 259320/557278 [00:05<00:10, 29383.69 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  47%|████▋     | 263320/557278 [00:05<00:09, 30014.61 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  48%|████▊     | 267320/557278 [00:06<00:09, 29395.47 examples/s]
Dro
1: pping Long Sequences (>16384) (num_proc=32):  49%|████▊     | 271320/557278 [00:06<00:11, 25926.52 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  49%|████▉     | 274320/557278 [00:06<00:10, 25846.33 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  50%|████▉     | 277320/557278 [00:06<00:10, 25865.02 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  50%|█████     | 281320/557278 [00:06<00:09, 28554.43 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  51%|█████▏    | 286320/557278 [00:06<00:08, 33281.06 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  52%|█████▏    | 290320/557278 [00:06<00:08, 30185.29 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  53%|█████▎    | 294320/557278 [00:07<00:08, 32103.81 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  54%|█████▎    | 298320/557278 [00:07<00:10, 23994.35 examples/s]
Dropping Long Sequences (>1
1: 6384) (num_proc=32):  54%|█████▍    | 303320/557278 [00:07<00:09, 28045.12 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  55%|█████▌    | 307320/557278 [00:07<00:08, 29525.94 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  56%|█████▌    | 312320/557278 [00:07<00:08, 29043.54 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  57%|█████▋    | 316320/557278 [00:07<00:08, 28896.52 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  57%|█████▋    | 320320/557278 [00:08<00:08, 26612.45 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  58%|█████▊    | 323320/557278 [00:08<00:10, 22826.62 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  59%|█████▉    | 329320/557278 [00:08<00:07, 30061.30 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  60%|█████▉    | 334320/557278 [00:08<00:06, 34299.36 examples/s]
Dropping Long Sequences (>16384) (num_proc=
1: 32):  61%|██████    | 338320/557278 [00:08<00:07, 27682.03 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  61%|██████▏   | 342320/557278 [00:08<00:07, 29127.76 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  62%|██████▏   | 346320/557278 [00:09<00:09, 21460.08 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  64%|██████▎   | 354320/557278 [00:09<00:06, 31728.40 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  64%|██████▍   | 359320/557278 [00:09<00:05, 33567.87 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  65%|██████▌   | 364320/557278 [00:09<00:06, 27800.85 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  66%|██████▌   | 368320/557278 [00:09<00:06, 27596.72 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  67%|██████▋   | 372320/557278 [00:09<00:07, 23861.72 examples/s]
Dropping Long Sequences (>16384) (num_proc=32
1: ):  68%|██████▊   | 379320/557278 [00:10<00:05, 30016.21 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  69%|██████▉   | 384320/557278 [00:10<00:05, 31094.78 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  70%|██████▉   | 388320/557278 [00:10<00:06, 26749.09 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  70%|███████   | 392320/557278 [00:10<00:05, 28252.45 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  71%|███████   | 396320/557278 [00:10<00:06, 23385.27 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  72%|███████▏  | 402320/557278 [00:10<00:05, 30025.43 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  73%|███████▎  | 408320/557278 [00:11<00:04, 32233.82 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  74%|███████▍  | 412320/557278 [00:11<00:05, 26247.60 examples/s]
Dropping Long Sequences (>16384) (num_p
1: roc=32):  75%|███████▍  | 417320/557278 [00:11<00:05, 26182.43 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  75%|███████▌  | 420320/557278 [00:11<00:05, 24670.99 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  77%|███████▋  | 426320/557278 [00:11<00:04, 30292.32 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  77%|███████▋  | 431320/557278 [00:11<00:04, 31486.76 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  78%|███████▊  | 435320/557278 [00:12<00:04, 25316.50 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  79%|███████▉  | 440320/557278 [00:12<00:03, 29507.36 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  80%|███████▉  | 444320/557278 [00:12<00:04, 23860.01 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  81%|████████  | 451320/557278 [00:12<00:03, 30778.12 examples/s]
Dropping Long Sequences
1:  (>16384) (num_proc=32):  82%|████████▏ | 455320/557278 [00:12<00:03, 32041.71 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  82%|████████▏ | 459320/557278 [00:13<00:03, 25271.46 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  83%|████████▎ | 464320/557278 [00:13<00:03, 27319.04 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  84%|████████▍ | 468320/557278 [00:13<00:03, 22917.31 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  85%|████████▌ | 475320/557278 [00:13<00:02, 31316.08 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  86%|████████▌ | 479320/557278 [00:13<00:02, 32194.13 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  87%|████████▋ | 483320/557278 [00:13<00:02, 25440.51 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  87%|████████▋ | 487320/557278 [00:13<00:02, 27511.70 exa
1: mples/s]
Dropping Long Sequences (>16384) (num_proc=32):  88%|████████▊ | 491320/557278 [00:14<00:02, 23383.26 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  89%|████████▉ | 497320/557278 [00:14<00:01, 30159.06 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  90%|█████████ | 502320/557278 [00:14<00:01, 33968.91 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  91%|█████████ | 507320/557278 [00:14<00:02, 24625.30 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  92%|█████████▏| 512320/557278 [00:14<00:01, 24127.26 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  93%|█████████▎| 516320/557278 [00:15<00:01, 25782.54 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  93%|█████████▎| 520320/557278 [00:15<00:01, 28152.89 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  95%|█████████▍| 
1: 527320/557278 [00:15<00:00, 34260.11 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  95%|█████████▌| 531150/557278 [00:15<00:00, 27189.62 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  96%|█████████▋| 536980/557278 [00:15<00:00, 32673.43 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  97%|█████████▋| 541225/557278 [00:15<00:00, 26300.73 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  98%|█████████▊| 545885/557278 [00:16<00:00, 28612.16 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  99%|█████████▊| 550300/557278 [00:16<00:00, 31492.97 examples/s]
Dropping Long Sequences (>16384) (num_proc=32):  99%|█████████▉| 553959/557278 [00:16<00:00, 29369.74 examples/s]
Dropping Long Sequences (>16384) (num_proc=32): 100%|██████████| 557278/557278 [00:16<00:00, 22953.72 examples/s]
Dropping Long Sequences (>16384) (num
1: _proc=32): 100%|██████████| 557278/557278 [00:16<00:00, 33385.54 examples/s]
1: 
Drop Samples with Zero Trainable Tokens (num_proc=32):   0%|          | 0/551276 [00:00<?, ? examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):   0%|          | 1000/551276 [00:00<04:44, 1934.20 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):   2%|▏         | 10000/551276 [00:00<00:31, 17365.00 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):   3%|▎         | 18000/551276 [00:00<00:21, 25222.84 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):   5%|▍         | 26000/551276 [00:01<00:17, 30462.90 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):   7%|▋         | 37000/551276 [00:01<00:11, 44983.52 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  11%|█         | 59000/551276 [00:01<00:06, 81914.98 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  13%|█▎        | 71000/551276 [00:01<00:06, 72551.54 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  15%|█▍
1:         | 81000/551276 [00:01<00:07, 63295.42 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  16%|█▌        | 89000/551276 [00:01<00:08, 56700.73 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  17%|█▋        | 96000/551276 [00:02<00:08, 56507.36 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  22%|██▏       | 121000/551276 [00:02<00:04, 96384.58 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  24%|██▍       | 134000/551276 [00:02<00:06, 67879.98 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  26%|██▌       | 144000/551276 [00:02<00:06, 62543.18 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  28%|██▊       | 153000/551276 [00:02<00:06, 59784.56 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  32%|███▏      | 177000/551276 [00:02<00:04, 86392.63 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  34%|███▍      | 1
1: 88000/551276 [00:03<00:04, 75013.74 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  36%|███▌      | 197000/551276 [00:03<00:05, 65079.15 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  37%|███▋      | 205000/551276 [00:03<00:05, 58058.91 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  40%|████      | 221824/551276 [00:03<00:04, 78166.35 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  42%|████▏     | 233824/551276 [00:03<00:03, 82214.60 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  44%|████▍     | 243824/551276 [00:04<00:08, 38400.37 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  46%|████▌     | 254824/551276 [00:04<00:06, 47130.76 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  48%|████▊     | 263824/551276 [00:05<00:09, 29219.27 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  49%|█�
1: �██▉     | 272824/551276 [00:05<00:07, 35338.73 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  51%|█████▏    | 282824/551276 [00:05<00:06, 40372.25 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  53%|█████▎    | 289824/551276 [00:06<00:09, 27184.12 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  54%|█████▍    | 299824/551276 [00:06<00:07, 34722.74 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  56%|█████▌    | 306824/551276 [00:06<00:06, 36998.72 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  57%|█████▋    | 312824/551276 [00:06<00:09, 24684.42 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  59%|█████▊    | 322824/551276 [00:06<00:06, 33528.92 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  60%|█████▉    | 329824/551276 [00:07<00:05, 38554.36 examples/s]
Drop Samples with Zero Trai
1: nable Tokens (num_proc=32):  61%|██████    | 336824/551276 [00:07<00:08, 24123.86 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  63%|██████▎   | 345824/551276 [00:07<00:06, 31534.11 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  64%|██████▍   | 353824/551276 [00:07<00:05, 37821.50 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  65%|██████▌   | 360824/551276 [00:08<00:07, 24101.53 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  67%|██████▋   | 370824/551276 [00:08<00:05, 31655.20 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  69%|██████▊   | 377824/551276 [00:08<00:04, 35923.46 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  70%|██████▉   | 383824/551276 [00:09<00:07, 23576.28 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  71%|███████▏  | 393824/551276 [00:09
1: <00:04, 32432.59 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  73%|███████▎  | 399824/551276 [00:09<00:04, 35369.01 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  74%|███████▎  | 405824/551276 [00:09<00:06, 24214.60 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  75%|███████▍  | 412824/551276 [00:10<00:04, 30054.17 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  76%|███████▌  | 418824/551276 [00:10<00:04, 31997.67 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  77%|███████▋  | 424824/551276 [00:10<00:03, 36584.81 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  78%|███████▊  | 430824/551276 [00:10<00:05, 23842.22 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  80%|███████▉  | 438824/551276 [00:10<00:03, 31525.43 examples/s]
Drop Samples with Zero Trainable Tok
1: ens (num_proc=32):  81%|████████  | 444824/551276 [00:11<00:03, 32619.49 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  82%|████████▏ | 449824/551276 [00:11<00:02, 34037.27 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  83%|████████▎ | 454824/551276 [00:11<00:03, 24700.01 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  84%|████████▍ | 461824/551276 [00:11<00:02, 30447.76 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  85%|████████▍ | 466824/551276 [00:11<00:02, 31440.00 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  86%|████████▌ | 473824/551276 [00:11<00:02, 33883.03 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  87%|████████▋ | 477824/551276 [00:12<00:02, 25207.23 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  88%|████████▊ |
1:  482824/551276 [00:12<00:02, 29027.51 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  88%|████████▊ | 487824/551276 [00:12<00:01, 31868.30 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  89%|████████▉ | 491824/551276 [00:12<00:01, 30842.29 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  90%|████████▉ | 495824/551276 [00:12<00:01, 32751.32 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  91%|█████████ | 499824/551276 [00:13<00:02, 23921.94 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  92%|█████████▏| 505824/551276 [00:13<00:01, 29857.55 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  93%|█████████▎| 510824/551276 [00:13<00:01, 33459.76 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  93%|█████████▎| 514824/551276 [00:13<00:01, 29283.02 example
1: s/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  94%|█████████▍| 518824/551276 [00:13<00:01, 31478.37 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  95%|█████████▍| 522824/551276 [00:13<00:00, 31489.76 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  96%|█████████▌| 527052/551276 [00:13<00:00, 28402.09 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  96%|█████████▋| 531052/551276 [00:13<00:00, 29373.32 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  97%|█████████▋| 536415/551276 [00:14<00:00, 34015.49 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  98%|█████████▊| 540323/551276 [00:14<00:00, 34141.78 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32):  99%|█████████▉| 544460/551276 [00:14<00:00, 33740.47 examples/s]
Drop Samples with Zero Trainable
1:  Tokens (num_proc=32): 100%|█████████▉| 548687/551276 [00:14<00:00, 34844.05 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=32): 100%|██████████| 551276/551276 [00:14<00:00, 37287.44 examples/s]
1: 
Add position_id column (Sample Packing) (num_proc=32):   0%|          | 0/551276 [00:00<?, ? examples/s]
Add position_id column (Sample Packing) (num_proc=32):   0%|          | 1000/551276 [00:00<05:47, 1584.75 examples/s]
Add position_id column (Sample Packing) (num_proc=32):   2%|▏         | 10000/551276 [00:00<00:42, 12636.13 examples/s]
Add position_id column (Sample Packing) (num_proc=32):   3%|▎         | 18000/551276 [00:01<00:30, 17365.69 examples/s]
Add position_id column (Sample Packing) (num_proc=32):   5%|▍         | 26000/551276 [00:01<00:26, 20017.41 examples/s]
Add position_id column (Sample Packing) (num_proc=32):   7%|▋         | 37000/551276 [00:01<00:16, 31792.60 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  10%|▉         | 55000/551276 [00:01<00:08, 55260.44 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  12%|█▏        | 65000/551276 [00:01<00:08, 54797.88 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  13%|█▎
1:         | 74000/551276 [00:02<00:14, 32998.96 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  15%|█▍        | 82000/551276 [00:02<00:15, 30498.12 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  16%|█▋        | 90000/551276 [00:03<00:15, 30727.36 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  20%|█▉        | 109000/551276 [00:03<00:08, 51192.64 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  22%|██▏       | 120000/551276 [00:03<00:07, 59092.35 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  24%|██▎       | 130000/551276 [00:03<00:10, 39978.16 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  25%|██▍       | 137000/551276 [00:03<00:09, 42791.34 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  26%|██▌       | 144000/551276 [00:04<00:11, 36226.68 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  27%|██▋       | 150000/
1: 551276 [00:04<00:12, 31838.96 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  30%|██▉       | 164000/551276 [00:04<00:08, 46653.59 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  32%|███▏      | 174000/551276 [00:04<00:07, 52229.86 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  33%|███▎      | 183000/551276 [00:04<00:06, 56376.92 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  35%|███▍      | 191000/551276 [00:05<00:08, 44782.33 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  36%|███▌      | 197000/551276 [00:05<00:09, 35958.20 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  37%|███▋      | 202000/551276 [00:05<00:11, 30473.66 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  38%|███▊      | 207456/551276 [00:05<00:10, 34109.90 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  39%|███▊      | 2
1: 12368/551276 [00:05<00:09, 35860.08 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  41%|████      | 224824/551276 [00:06<00:06, 52640.83 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  42%|████▏     | 231824/551276 [00:06<00:06, 51140.78 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  43%|████▎     | 237824/551276 [00:07<00:19, 15941.37 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  45%|████▍     | 245824/551276 [00:07<00:14, 21414.56 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  46%|████▌     | 254824/551276 [00:07<00:10, 28175.38 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  47%|████▋     | 260824/551276 [00:08<00:21, 13664.06 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  49%|████▉     | 270824/551276 [00:08<00:14, 19625.81 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  50%|�
1: ��████     | 277824/551276 [00:08<00:11, 23845.56 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  51%|█████▏    | 283824/551276 [00:09<00:19, 13490.01 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  53%|█████▎    | 289824/551276 [00:10<00:15, 16637.58 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  54%|█████▍    | 296824/551276 [00:10<00:11, 21354.51 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  55%|█████▌    | 303824/551276 [00:10<00:09, 26243.87 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  56%|█████▌    | 309824/551276 [00:11<00:17, 13556.72 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  57%|█████▋    | 313824/551276 [00:11<00:15, 15276.07 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  58%|█████▊    | 320824/551276 [00:11<00:11, 20517.09 examples/s]
Add position_id column 
1: (Sample Packing) (num_proc=32):  59%|█████▉    | 326824/551276 [00:11<00:08, 25152.23 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  60%|██████    | 331824/551276 [00:12<00:16, 13364.13 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  61%|██████    | 335824/551276 [00:12<00:14, 14799.78 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  62%|██████▏   | 339824/551276 [00:12<00:12, 17346.23 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  63%|██████▎   | 348824/551276 [00:12<00:07, 25846.75 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  64%|██████▍   | 353824/551276 [00:13<00:07, 26053.63 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  65%|██████▍   | 357824/551276 [00:14<00:15, 12680.87 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  65%|██████▌   | 360824/551276 [00:14<0
1: 0:13, 14266.14 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  67%|██████▋   | 366824/551276 [00:14<00:09, 19555.09 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  68%|██████▊   | 374824/551276 [00:14<00:07, 25158.08 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  69%|██████▊   | 378824/551276 [00:14<00:08, 20743.77 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  69%|██████▉   | 382824/551276 [00:15<00:13, 12208.04 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  70%|███████   | 387824/551276 [00:15<00:10, 15402.89 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  72%|███████▏  | 396824/551276 [00:15<00:06, 23944.61 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  73%|███████▎  | 401824/551276 [00:15<00:06, 22427.41 examples/s]
Add position_id column (Sample Packing) (num_pro
1: c=32):  74%|███████▎  | 405824/551276 [00:16<00:11, 12611.48 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  74%|███████▍  | 409824/551276 [00:16<00:09, 14435.04 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  76%|███████▌  | 416824/551276 [00:17<00:06, 20539.55 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  77%|███████▋  | 421824/551276 [00:17<00:05, 24224.84 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  77%|███████▋  | 426824/551276 [00:17<00:07, 17524.76 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  78%|███████▊  | 430824/551276 [00:18<00:09, 12968.39 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  79%|███████▉  | 436824/551276 [00:18<00:06, 17242.75 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  80%|████████  | 442824/551276 [00:18<00:0
1: 5, 21457.20 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  81%|████████  | 446824/551276 [00:18<00:04, 23763.51 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  82%|████████▏ | 450824/551276 [00:18<00:05, 18348.17 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  82%|████████▏ | 453824/551276 [00:19<00:08, 11826.72 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  83%|████████▎ | 456824/551276 [00:19<00:06, 13613.69 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  84%|████████▍ | 462824/551276 [00:19<00:04, 19546.29 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  85%|████████▍ | 466824/551276 [00:19<00:03, 22478.04 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  85%|████████▌ | 470824/551276 [00:19<00:03, 25196.92 examples/s]
Add position_id column (Sampl
1: e Packing) (num_proc=32):  86%|████████▌ | 474824/551276 [00:20<00:04, 17243.95 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  87%|████████▋ | 477824/551276 [00:20<00:06, 11768.38 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  87%|████████▋ | 481824/551276 [00:21<00:04, 14205.39 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  88%|████████▊ | 486824/551276 [00:21<00:03, 19055.21 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  89%|████████▉ | 492824/551276 [00:21<00:02, 22058.92 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  90%|█████████ | 496824/551276 [00:21<00:02, 23027.45 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  91%|█████████ | 499824/551276 [00:21<00:03, 16588.37 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  91%|██████�
1: �██ | 502824/551276 [00:22<00:03, 13301.27 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  92%|█████████▏| 504824/551276 [00:22<00:03, 13444.74 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  93%|█████████▎| 510824/551276 [00:22<00:02, 19727.22 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  94%|█████████▍| 516824/551276 [00:22<00:01, 23079.40 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  94%|█████████▍| 520824/551276 [00:22<00:01, 24051.01 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  95%|█████████▌| 524052/551276 [00:23<00:01, 16179.16 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  95%|█████████▌| 526279/551276 [00:23<00:02, 12452.67 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  96%|█████████▌| 528506/551276 [00:23<00:01,
1:  13466.04 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  96%|█████████▋| 531506/551276 [00:23<00:01, 15329.76 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  98%|█████████▊| 539960/551276 [00:23<00:00, 27106.60 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  99%|█████████▊| 543777/551276 [00:24<00:00, 21908.53 examples/s]
Add position_id column (Sample Packing) (num_proc=32):  99%|█████████▉| 547913/551276 [00:24<00:00, 24700.09 examples/s]
Add position_id column (Sample Packing) (num_proc=32): 100%|██████████| 551276/551276 [00:24<00:00, 16356.03 examples/s]
Add position_id column (Sample Packing) (num_proc=32): 100%|██████████| 551276/551276 [00:24<00:00, 22155.62 examples/s]
1: 
Saving the dataset (0/45 shards):   0%|          | 0/551276 [00:00<?, ? examples/s]
Saving the dataset (0/45 shards):   1%|          | 5000/551276 [00:00<00:40, 13494.40 examples/s]
Saving the dataset (1/45 shards):  18%|█▊        | 97251/551276 [00:00<00:33, 13494.40 examples/s]
Saving the dataset (1/45 shards):  22%|██▏       | 119251/551276 [00:00<00:01, 326466.61 examples/s]
Saving the dataset (2/45 shards):  24%|██▎       | 130502/551276 [00:00<00:01, 326466.61 examples/s]
Saving the dataset (3/45 shards):  26%|██▌       | 140753/551276 [00:00<00:01, 326466.61 examples/s]
Saving the dataset (4/45 shards):  28%|██▊       | 157004/551276 [00:00<00:01, 326466.61 examples/s]
Saving the dataset (5/45 shards):  29%|██▉       | 161255/551276 [00:00<00:01, 326466.61 examples/s]
Saving the dataset (6/45 shards):  31%|███       | 168757/551276 [00:00<00:01, 326466.61 examples/s]
Saving the dataset (7/45 shards):  31%|███       | 170008/551276 [00:00<00:01, 326466.61 e
1: xamples/s]
Saving the dataset (8/45 shards):  31%|███▏      | 173259/551276 [00:00<00:01, 326466.61 examples/s]
Saving the dataset (9/45 shards):  32%|███▏      | 175510/551276 [00:00<00:01, 326466.61 examples/s]
Saving the dataset (10/45 shards):  32%|███▏      | 176510/551276 [00:00<00:01, 326466.61 examples/s]
Saving the dataset (11/45 shards):  33%|███▎      | 180761/551276 [00:00<00:01, 326466.61 examples/s]
Saving the dataset (11/45 shards):  34%|███▍      | 189761/551276 [00:00<00:00, 422757.41 examples/s]
Saving the dataset (11/45 shards):  46%|████▋     | 255761/551276 [00:00<00:00, 426461.07 examples/s]
Saving the dataset (11/45 shards):  57%|█████▋    | 312761/551276 [00:00<00:00, 445037.91 examples/s]
Saving the dataset (12/45 shards):  65%|██████▍   | 358012/551276 [00:00<00:00, 445037.91 examples/s]
Saving the dataset (12/45 shards):  67%|██████▋   | 367012/551276 [00:00<00:00, 427326.71 examples/s]
Saving 
1: the dataset (12/45 shards):  75%|███████▌  | 416012/551276 [00:01<00:00, 435904.61 examples/s]
Saving the dataset (13/45 shards):  81%|████████▏ | 449263/551276 [00:01<00:00, 435904.61 examples/s]
Saving the dataset (14/45 shards):  84%|████████▎ | 461514/551276 [00:01<00:00, 435904.61 examples/s]
Saving the dataset (14/45 shards):  84%|████████▍ | 464514/551276 [00:01<00:00, 436057.34 examples/s]
Saving the dataset (15/45 shards):  86%|████████▌ | 472765/551276 [00:01<00:00, 436057.34 examples/s]
Saving the dataset (16/45 shards):  86%|████████▌ | 475016/551276 [00:01<00:00, 436057.34 examples/s]
Saving the dataset (17/45 shards):  86%|████████▋ | 476267/551276 [00:01<00:00, 436057.34 examples/s]
Saving the dataset (18/45 shards):  87%|████████▋ | 477518/551276 [00:01<00:00, 436057.34 examples/s]
Saving the dataset (19/45 shards):  87%|████████▋ | 478
1: 769/551276 [00:01<00:00, 436057.34 examples/s]
Saving the dataset (20/45 shards):  88%|████████▊ | 485020/551276 [00:01<00:00, 436057.34 examples/s]
Saving the dataset (21/45 shards):  89%|████████▉ | 489270/551276 [00:01<00:00, 436057.34 examples/s]
Saving the dataset (22/45 shards):  89%|████████▉ | 489521/551276 [00:01<00:00, 436057.34 examples/s]
Saving the dataset (23/45 shards):  89%|████████▉ | 490772/551276 [00:01<00:00, 436057.34 examples/s]
Saving the dataset (24/45 shards):  89%|████████▉ | 492023/551276 [00:01<00:00, 436057.34 examples/s]
Saving the dataset (25/45 shards):  89%|████████▉ | 493273/551276 [00:01<00:00, 436057.34 examples/s]
Saving the dataset (26/45 shards):  90%|████████▉ | 493524/551276 [00:01<00:00, 436057.34 examples/s]
Saving the dataset (27/45 shards):  90%|█████████ | 497775/551276 [00:01<00:00, 436057.34 examples/s]
Saving the datase
1: t (28/45 shards):  91%|█████████ | 502026/551276 [00:01<00:00, 436057.34 examples/s]
Saving the dataset (29/45 shards):  91%|█████████▏| 503276/551276 [00:01<00:00, 436057.34 examples/s]
Saving the dataset (30/45 shards):  92%|█████████▏| 504526/551276 [00:01<00:00, 436057.34 examples/s]
Saving the dataset (31/45 shards):  92%|█████████▏| 506776/551276 [00:01<00:00, 436057.34 examples/s]
Saving the dataset (31/45 shards):  93%|█████████▎| 511776/551276 [00:01<00:00, 420241.55 examples/s]
Saving the dataset (32/45 shards):  93%|█████████▎| 513026/551276 [00:01<00:00, 420241.55 examples/s]
Saving the dataset (33/45 shards):  94%|█████████▍| 517276/551276 [00:01<00:00, 420241.55 examples/s]
Saving the dataset (34/45 shards):  94%|█████████▍| 519526/551276 [00:01<00:00, 420241.55 examples/s]
Saving the dataset (35/45 shards):  95%|█████████
1: ▌| 523776/551276 [00:01<00:00, 420241.55 examples/s]
Saving the dataset (36/45 shards):  96%|█████████▌| 527026/551276 [00:01<00:00, 420241.55 examples/s]
Saving the dataset (37/45 shards):  96%|█████████▌| 529276/551276 [00:01<00:00, 420241.55 examples/s]
Saving the dataset (38/45 shards):  96%|█████████▋| 531526/551276 [00:01<00:00, 420241.55 examples/s]
Saving the dataset (39/45 shards):  96%|█████████▋| 531776/551276 [00:01<00:00, 420241.55 examples/s]
Saving the dataset (40/45 shards):  97%|█████████▋| 532026/551276 [00:01<00:00, 420241.55 examples/s]
Saving the dataset (41/45 shards):  97%|█████████▋| 532276/551276 [00:01<00:00, 420241.55 examples/s]
Saving the dataset (42/45 shards):  97%|█████████▋| 536776/551276 [00:01<00:00, 420241.55 examples/s]
Saving the dataset (43/45 shards):  97%|█████████▋| 536776/551276 [00:01<00:00, 420241.55 examp
1: les/s]
Saving the dataset (44/45 shards):  99%|█████████▉| 548026/551276 [00:01<00:00, 420241.55 examples/s]
Saving the dataset (45/45 shards): 100%|██████████| 551276/551276 [00:01<00:00, 420241.55 examples/s]
Saving the dataset (45/45 shards): 100%|██████████| 551276/551276 [00:01<00:00, 278978.01 examples/s]
0: [2025-11-23 14:45:42,169] [INFO] [axolotl.utils.data.shared.load_preprocessed_dataset:472] [PID:2356327] [RANK:0] Loading prepared dataset from disk at /lustre/fswork/projects/rech/dgo/udv55np/dataset_gemma/Nemotron-Super-49B-v1_5/split_0.75/d383a692316f2e59f08e99509bd12afa...
0: jzxh014:2356327:2356327 [0] NCCL INFO Bootstrap : Using ibp24s0:10.100.4.53<0>
0: jzxh014:2356327:2356327 [0] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
0: jzxh014:2356327:2356327 [0] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
0: jzxh014:2356327:2356327 [0] NCCL INFO NET/Plugin: Using internal network plugin.
0: jzxh014:2356327:2356327 [0] NCCL INFO cudaDriverVersion 12080
0: NCCL version 2.21.5+cuda12.4
0: jzxh014:2356327:2356327 [0] NCCL INFO Comm config Blocking set to 1
0: jzxh014:2356329:2356329 [2] NCCL INFO cudaDriverVersion 12080
0: jzxh014:2356329:2356329 [2] NCCL INFO Bootstrap : Using ibp24s0:10.100.4.53<0>
0: jzxh014:2356329:2356329 [2] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
0: jzxh014:2356329:2356329 [2] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
0: jzxh014:2356329:2356329 [2] NCCL INFO NET/Plugin: Using internal network plugin.
0: jzxh014:2356329:2356329 [2] NCCL INFO Comm config Blocking set to 1
0: jzxh014:2356328:2356328 [1] NCCL INFO cudaDriverVersion 12080
0: jzxh014:2356330:2356330 [3] NCCL INFO cudaDriverVersion 12080
1: jzxh015:2965582:2965582 [0] NCCL INFO cudaDriverVersion 12080
3: jzxh017:51583:51583 [1] NCCL INFO cudaDriverVersion 12080
0: jzxh014:2356328:2356328 [1] NCCL INFO Bootstrap : Using ibp24s0:10.100.4.53<0>
0: jzxh014:2356330:2356330 [3] NCCL INFO Bootstrap : Using ibp24s0:10.100.4.53<0>
0: jzxh014:2356330:2356330 [3] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
0: jzxh014:2356328:2356328 [1] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
0: jzxh014:2356330:2356330 [3] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
0: jzxh014:2356330:2356330 [3] NCCL INFO NET/Plugin: Using internal network plugin.
0: jzxh014:2356328:2356328 [1] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
0: jzxh014:2356328:2356328 [1] NCCL INFO NET/Plugin: Using internal network plugin.
0: jzxh014:2356330:2356330 [3] NCCL INFO Comm config Blocking set to 1
0: jzxh014:2356328:2356328 [1] NCCL INFO Comm config Blocking set to 1
1: jzxh015:2965582:2965582 [0] NCCL INFO Bootstrap : Using ibp24s0:10.100.4.57<0>
3: jzxh017:51583:51583 [1] NCCL INFO Bootstrap : Using ibp24s0:10.100.4.65<0>
1: jzxh015:2965583:2965583 [1] NCCL INFO cudaDriverVersion 12080
3: jzxh017:51582:51582 [0] NCCL INFO cudaDriverVersion 12080
3: jzxh017:51582:51582 [0] NCCL INFO Bootstrap : Using ibp24s0:10.100.4.65<0>
2: jzxh016:896066:896066 [2] NCCL INFO cudaDriverVersion 12080
2: jzxh016:896066:896066 [2] NCCL INFO Bootstrap : Using ibp24s0:10.100.4.61<0>
1: jzxh015:2965584:2965584 [2] NCCL INFO cudaDriverVersion 12080
1: jzxh015:2965583:2965583 [1] NCCL INFO Bootstrap : Using ibp24s0:10.100.4.57<0>
3: jzxh017:51582:51582 [0] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
3: jzxh017:51583:51583 [1] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
1: jzxh015:2965584:2965584 [2] NCCL INFO Bootstrap : Using ibp24s0:10.100.4.57<0>
3: jzxh017:51583:51583 [1] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
3: jzxh017:51582:51582 [0] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
3: jzxh017:51583:51583 [1] NCCL INFO NET/Plugin: Using internal network plugin.
1: jzxh015:2965583:2965583 [1] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
1: jzxh015:2965582:2965582 [0] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
3: jzxh017:51582:51582 [0] NCCL INFO NET/Plugin: Using internal network plugin.
1: jzxh015:2965583:2965583 [1] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
1: jzxh015:2965583:2965583 [1] NCCL INFO NET/Plugin: Using internal network plugin.
1: jzxh015:2965582:2965582 [0] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
1: jzxh015:2965582:2965582 [0] NCCL INFO NET/Plugin: Using internal network plugin.
1: jzxh015:2965584:2965584 [2] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
1: jzxh015:2965584:2965584 [2] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
1: jzxh015:2965584:2965584 [2] NCCL INFO NET/Plugin: Using internal network plugin.
1: jzxh015:2965585:2965585 [3] NCCL INFO cudaDriverVersion 12080
1: jzxh015:2965585:2965585 [3] NCCL INFO Bootstrap : Using ibp24s0:10.100.4.57<0>
1: jzxh015:2965585:2965585 [3] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
2: jzxh016:896064:896064 [0] NCCL INFO cudaDriverVersion 12080
1: jzxh015:2965585:2965585 [3] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
1: jzxh015:2965585:2965585 [3] NCCL INFO NET/Plugin: Using internal network plugin.
1: jzxh015:2965582:2965582 [0] NCCL INFO Comm config Blocking set to 1
1: jzxh015:2965583:2965583 [1] NCCL INFO Comm config Blocking set to 1
2: jzxh016:896066:896066 [2] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
2: jzxh016:896066:896066 [2] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
2: jzxh016:896066:896066 [2] NCCL INFO NET/Plugin: Using internal network plugin.
1: jzxh015:2965584:2965584 [2] NCCL INFO Comm config Blocking set to 1
3: jzxh017:51584:51584 [2] NCCL INFO cudaDriverVersion 12080
3: jzxh017:51583:51583 [1] NCCL INFO Comm config Blocking set to 1
3: jzxh017:51582:51582 [0] NCCL INFO Comm config Blocking set to 1
2: jzxh016:896064:896064 [0] NCCL INFO Bootstrap : Using ibp24s0:10.100.4.61<0>
3: jzxh017:51585:51585 [3] NCCL INFO cudaDriverVersion 12080
2: jzxh016:896064:896064 [0] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
2: jzxh016:896064:896064 [0] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
2: jzxh016:896064:896064 [0] NCCL INFO NET/Plugin: Using internal network plugin.
1: jzxh015:2965585:2965585 [3] NCCL INFO Comm config Blocking set to 1
3: jzxh017:51584:51584 [2] NCCL INFO Bootstrap : Using ibp24s0:10.100.4.65<0>
3: jzxh017:51585:51585 [3] NCCL INFO Bootstrap : Using ibp24s0:10.100.4.65<0>
3: jzxh017:51585:51585 [3] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
3: jzxh017:51585:51585 [3] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
3: jzxh017:51585:51585 [3] NCCL INFO NET/Plugin: Using internal network plugin.
3: jzxh017:51584:51584 [2] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
3: jzxh017:51584:51584 [2] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
3: jzxh017:51584:51584 [2] NCCL INFO NET/Plugin: Using internal network plugin.
2: jzxh016:896067:896067 [3] NCCL INFO cudaDriverVersion 12080
2: jzxh016:896066:896066 [2] NCCL INFO Comm config Blocking set to 1
2: jzxh016:896064:896064 [0] NCCL INFO Comm config Blocking set to 1
2: jzxh016:896067:896067 [3] NCCL INFO Bootstrap : Using ibp24s0:10.100.4.61<0>
2: jzxh016:896067:896067 [3] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
3: jzxh017:51585:51585 [3] NCCL INFO Comm config Blocking set to 1
3: jzxh017:51584:51584 [2] NCCL INFO Comm config Blocking set to 1
2: jzxh016:896067:896067 [3] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
2: jzxh016:896067:896067 [3] NCCL INFO NET/Plugin: Using internal network plugin.
2: jzxh016:896065:896065 [1] NCCL INFO cudaDriverVersion 12080
2: jzxh016:896065:896065 [1] NCCL INFO Bootstrap : Using ibp24s0:10.100.4.61<0>
2: jzxh016:896067:896067 [3] NCCL INFO Comm config Blocking set to 1
2: jzxh016:896065:896065 [1] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
2: jzxh016:896065:896065 [1] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
2: jzxh016:896065:896065 [1] NCCL INFO NET/Plugin: Using internal network plugin.
2: jzxh016:896065:896065 [1] NCCL INFO Comm config Blocking set to 1
0: jzxh014:2356329:2357228 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.4.53<0>
0: jzxh014:2356328:2357230 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.4.53<0>
0: jzxh014:2356329:2357228 [2] NCCL INFO Using non-device net plugin version 0
0: jzxh014:2356329:2357228 [2] NCCL INFO Using network IB
0: jzxh014:2356327:2357227 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.4.53<0>
0: jzxh014:2356328:2357230 [1] NCCL INFO Using non-device net plugin version 0
0: jzxh014:2356328:2357230 [1] NCCL INFO Using network IB
0: jzxh014:2356330:2357229 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.4.53<0>
0: jzxh014:2356327:2357227 [0] NCCL INFO Using non-device net plugin version 0
0: jzxh014:2356327:2357227 [0] NCCL INFO Using network IB
0: jzxh014:2356330:2357229 [3] NCCL INFO Using non-device net plugin version 0
0: jzxh014:2356330:2357229 [3] NCCL INFO Using network IB
1: jzxh015:2965583:2967788 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.4.57<0>
1: jzxh015:2965585:2967786 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.4.57<0>
1: jzxh015:2965584:2967789 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.4.57<0>
1: jzxh015:2965583:2967788 [1] NCCL INFO Using non-device net plugin version 0
1: jzxh015:2965583:2967788 [1] NCCL INFO Using network IB
1: jzxh015:2965582:2967787 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.4.57<0>
1: jzxh015:2965585:2967786 [3] NCCL INFO Using non-device net plugin version 0
1: jzxh015:2965585:2967786 [3] NCCL INFO Using network IB
1: jzxh015:2965584:2967789 [2] NCCL INFO Using non-device net plugin version 0
1: jzxh015:2965584:2967789 [2] NCCL INFO Using network IB
1: jzxh015:2965582:2967787 [0] NCCL INFO Using non-device net plugin version 0
1: jzxh015:2965582:2967787 [0] NCCL INFO Using network IB
2: jzxh016:896065:896964 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.4.61<0>
2: jzxh016:896067:896963 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.4.61<0>
2: jzxh016:896064:896962 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.4.61<0>
2: jzxh016:896065:896964 [1] NCCL INFO Using non-device net plugin version 0
2: jzxh016:896065:896964 [1] NCCL INFO Using network IB
2: jzxh016:896066:896961 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.4.61<0>
2: jzxh016:896067:896963 [3] NCCL INFO Using non-device net plugin version 0
2: jzxh016:896067:896963 [3] NCCL INFO Using network IB
2: jzxh016:896064:896962 [0] NCCL INFO Using non-device net plugin version 0
2: jzxh016:896064:896962 [0] NCCL INFO Using network IB
2: jzxh016:896066:896961 [2] NCCL INFO Using non-device net plugin version 0
2: jzxh016:896066:896961 [2] NCCL INFO Using network IB
3: jzxh017:51582:52477 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.4.65<0>
3: jzxh017:51585:52478 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.4.65<0>
3: jzxh017:51584:52479 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.4.65<0>
3: jzxh017:51582:52477 [0] NCCL INFO Using non-device net plugin version 0
3: jzxh017:51582:52477 [0] NCCL INFO Using network IB
3: jzxh017:51583:52476 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ibp24s0:10.100.4.65<0>
3: jzxh017:51585:52478 [3] NCCL INFO Using non-device net plugin version 0
3: jzxh017:51585:52478 [3] NCCL INFO Using network IB
3: jzxh017:51584:52479 [2] NCCL INFO Using non-device net plugin version 0
3: jzxh017:51584:52479 [2] NCCL INFO Using network IB
3: jzxh017:51583:52476 [1] NCCL INFO Using non-device net plugin version 0
3: jzxh017:51583:52476 [1] NCCL INFO Using network IB
0: jzxh014:2356330:2357229 [3] NCCL INFO ncclCommInitRank comm 0x55abd54283f0 rank 3 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x4261a6b6ec5d7236 - Init START
0: jzxh014:2356329:2357228 [2] NCCL INFO ncclCommInitRank comm 0x55e468be3960 rank 2 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x4261a6b6ec5d7236 - Init START
1: jzxh015:2965582:2967787 [0] NCCL INFO ncclCommInitRank comm 0x5581ebd8acb0 rank 4 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x4261a6b6ec5d7236 - Init START
1: jzxh015:2965583:2967788 [1] NCCL INFO ncclCommInitRank comm 0x55fffbc332b0 rank 5 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x4261a6b6ec5d7236 - Init START
0: jzxh014:2356328:2357230 [1] NCCL INFO ncclCommInitRank comm 0x559d12187a60 rank 1 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x4261a6b6ec5d7236 - Init START
0: jzxh014:2356327:2357227 [0] NCCL INFO ncclCommInitRank comm 0x555ba8ca5800 rank 0 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x4261a6b6ec5d7236 - Init START
2: jzxh016:896064:896962 [0] NCCL INFO ncclCommInitRank comm 0x55a50cad5640 rank 8 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x4261a6b6ec5d7236 - Init START
1: jzxh015:2965585:2967786 [3] NCCL INFO ncclCommInitRank comm 0x55ee7f078d80 rank 7 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x4261a6b6ec5d7236 - Init START
1: jzxh015:2965584:2967789 [2] NCCL INFO ncclCommInitRank comm 0x564e0a919f70 rank 6 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x4261a6b6ec5d7236 - Init START
3: jzxh017:51582:52477 [0] NCCL INFO ncclCommInitRank comm 0x559e3dba4770 rank 12 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x4261a6b6ec5d7236 - Init START
3: jzxh017:51583:52476 [1] NCCL INFO ncclCommInitRank comm 0x558abda268d0 rank 13 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x4261a6b6ec5d7236 - Init START
3: jzxh017:51584:52479 [2] NCCL INFO ncclCommInitRank comm 0x564cbfab3310 rank 14 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x4261a6b6ec5d7236 - Init START
3: jzxh017:51585:52478 [3] NCCL INFO ncclCommInitRank comm 0x56086fae1720 rank 15 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x4261a6b6ec5d7236 - Init START
2: jzxh016:896065:896964 [1] NCCL INFO ncclCommInitRank comm 0x55a46b9ae900 rank 9 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x4261a6b6ec5d7236 - Init START
2: jzxh016:896067:896963 [3] NCCL INFO ncclCommInitRank comm 0x5576b8f9bbe0 rank 11 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x4261a6b6ec5d7236 - Init START
2: jzxh016:896066:896961 [2] NCCL INFO ncclCommInitRank comm 0x556ee6361e10 rank 10 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x4261a6b6ec5d7236 - Init START
2: jzxh016:896067:896963 [3] NCCL INFO Setting affinity for GPU 3 to ffffff00,00000000,00000000,ffffff00,00000000,00000000
2: jzxh016:896067:896963 [3] NCCL INFO NVLS multicast support is not available on dev 3
2: jzxh016:896066:896961 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffff0000,00000000,000000ff,ffff0000,00000000
2: jzxh016:896066:896961 [2] NCCL INFO NVLS multicast support is not available on dev 2
2: jzxh016:896064:896962 [0] NCCL INFO Setting affinity for GPU 0 to ffffff,00000000,00000000,00ffffff
2: jzxh016:896064:896962 [0] NCCL INFO NVLS multicast support is not available on dev 0
2: jzxh016:896065:896964 [1] NCCL INFO Setting affinity for GPU 1 to ffff,ff000000,00000000,0000ffff,ff000000
2: jzxh016:896065:896964 [1] NCCL INFO NVLS multicast support is not available on dev 1
3: jzxh017:51582:52477 [0] NCCL INFO Setting affinity for GPU 0 to ffffff,00000000,00000000,00ffffff
3: jzxh017:51582:52477 [0] NCCL INFO NVLS multicast support is not available on dev 0
3: jzxh017:51583:52476 [1] NCCL INFO Setting affinity for GPU 1 to ffff,ff000000,00000000,0000ffff,ff000000
3: jzxh017:51583:52476 [1] NCCL INFO NVLS multicast support is not available on dev 1
0: jzxh014:2356327:2357227 [0] NCCL INFO Setting affinity for GPU 0 to ffffff,00000000,00000000,00ffffff
0: jzxh014:2356327:2357227 [0] NCCL INFO NVLS multicast support is not available on dev 0
3: jzxh017:51584:52479 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffff0000,00000000,000000ff,ffff0000,00000000
3: jzxh017:51584:52479 [2] NCCL INFO NVLS multicast support is not available on dev 2
0: jzxh014:2356330:2357229 [3] NCCL INFO Setting affinity for GPU 3 to ffffff00,00000000,00000000,ffffff00,00000000,00000000
0: jzxh014:2356330:2357229 [3] NCCL INFO NVLS multicast support is not available on dev 3
3: jzxh017:51585:52478 [3] NCCL INFO Setting affinity for GPU 3 to ffffff00,00000000,00000000,ffffff00,00000000,00000000
3: jzxh017:51585:52478 [3] NCCL INFO NVLS multicast support is not available on dev 3
1: jzxh015:2965584:2967789 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffff0000,00000000,000000ff,ffff0000,00000000
1: jzxh015:2965584:2967789 [2] NCCL INFO NVLS multicast support is not available on dev 2
0: jzxh014:2356329:2357228 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffff0000,00000000,000000ff,ffff0000,00000000
0: jzxh014:2356329:2357228 [2] NCCL INFO NVLS multicast support is not available on dev 2
0: jzxh014:2356328:2357230 [1] NCCL INFO Setting affinity for GPU 1 to ffff,ff000000,00000000,0000ffff,ff000000
0: jzxh014:2356328:2357230 [1] NCCL INFO NVLS multicast support is not available on dev 1
1: jzxh015:2965583:2967788 [1] NCCL INFO Setting affinity for GPU 1 to ffff,ff000000,00000000,0000ffff,ff000000
1: jzxh015:2965583:2967788 [1] NCCL INFO NVLS multicast support is not available on dev 1
1: jzxh015:2965582:2967787 [0] NCCL INFO Setting affinity for GPU 0 to ffffff,00000000,00000000,00ffffff
1: jzxh015:2965582:2967787 [0] NCCL INFO NVLS multicast support is not available on dev 0
1: jzxh015:2965585:2967786 [3] NCCL INFO Setting affinity for GPU 3 to ffffff00,00000000,00000000,ffffff00,00000000,00000000
1: jzxh015:2965585:2967786 [3] NCCL INFO NVLS multicast support is not available on dev 3
0: jzxh014:2356327:2357227 [0] NCCL INFO comm 0x555ba8ca5800 rank 0 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 00/16 :    0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 01/16 :    0   3   2   5   4   7   6   9   8  11  10  13  12  15  14   1
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 02/16 :    0   3   6   5   4   7  10   9   8  11  14  13  12  15   2   1
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 03/16 :    0   1   2   7   4   5   6  11   8   9  10  15  12  13  14   3
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 04/16 :    0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 05/16 :    0   3   2   5   4   7   6   9   8  11  10  13  12  15  14   1
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 06/16 :    0   3   6   5   4   7  10   9   8  11  14  13  12  15   2   1
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 07/16 :    0   1   2   7   4   5   6  11   8   9  10  15  12  13  14   3
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 08/16 :    0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 09/16 :    0   3   2   5   4   7   6   9   8  11  10  13  12  15  14   1
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 10/16 :    0   3   6   5   4   7  10   9   8  11  14  13  12  15   2   1
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 11/16 :    0   1   2   7   4   5   6  11   8   9  10  15  12  13  14   3
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 12/16 :    0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 13/16 :    0   3   2   5   4   7   6   9   8  11  10  13  12  15  14   1
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 14/16 :    0   3   6   5   4   7  10   9   8  11  14  13  12  15   2   1
0: jzxh014:2356328:2357230 [1] NCCL INFO comm 0x559d12187a60 rank 1 nRanks 16 nNodes 4 localRanks 4 localRank 1 MNNVL 0
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 15/16 :    0   1   2   7   4   5   6  11   8   9  10  15  12  13  14   3
0: jzxh014:2356330:2357229 [3] NCCL INFO comm 0x55abd54283f0 rank 3 nRanks 16 nNodes 4 localRanks 4 localRank 3 MNNVL 0
0: jzxh014:2356327:2357227 [0] NCCL INFO Trees [0] 1/8/-1->0->-1 [1] -1/-1/-1->0->3 [2] 1/-1/-1->0->2 [3] 2/-1/-1->0->1 [4] 3/8/-1->0->-1 [5] 2/-1/-1->0->3 [6] 3/-1/-1->0->2 [7] -1/-1/-1->0->1 [8] 1/-1/-1->0->4 [9] -1/-1/-1->0->3 [10] 1/-1/-1->0->2 [11] 2/-1/-1->0->1 [12] 3/-1/-1->0->4 [13] 2/-1/-1->0->3 [14] 3/-1/-1->0->2 [15] -1/-1/-1->0->1
0: jzxh014:2356327:2357227 [0] NCCL INFO P2P Chunksize set to 131072
3: jzxh017:51585:52478 [3] NCCL INFO comm 0x56086fae1720 rank 15 nRanks 16 nNodes 4 localRanks 4 localRank 3 MNNVL 0
3: jzxh017:51585:52478 [3] NCCL INFO Trees [0] -1/-1/-1->15->14 [1] 12/-1/-1->15->14 [2] -1/-1/-1->15->13 [3] 13/-1/-1->15->11 [4] 14/-1/-1->15->12 [5] 12/-1/-1->15->13 [6] 13/-1/-1->15->12 [7] 14/-1/-1->15->11 [8] -1/-1/-1->15->14 [9] 12/-1/-1->15->14 [10] -1/-1/-1->15->13 [11] 13/7/-1->15->-1 [12] 14/-1/-1->15->12 [13] 12/-1/-1->15->13 [14] 13/-1/-1->15->12 [15] 14/7/-1->15->-1
3: jzxh017:51585:52478 [3] NCCL INFO P2P Chunksize set to 131072
3: jzxh017:51584:52479 [2] NCCL INFO comm 0x564cbfab3310 rank 14 nRanks 16 nNodes 4 localRanks 4 localRank 2 MNNVL 0
0: jzxh014:2356328:2357230 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/9/-1->1->-1 [2] 3/-1/-1->1->0 [3] 0/-1/-1->1->3 [4] -1/-1/-1->1->2 [5] 3/9/-1->1->-1 [6] -1/-1/-1->1->3 [7] 0/-1/-1->1->2 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->5 [10] 3/-1/-1->1->0 [11] 0/-1/-1->1->3 [12] -1/-1/-1->1->2 [13] 3/-1/-1->1->5 [14] -1/-1/-1->1->3 [15] 0/-1/-1->1->2
0: jzxh014:2356329:2357228 [2] NCCL INFO comm 0x55e468be3960 rank 2 nRanks 16 nNodes 4 localRanks 4 localRank 2 MNNVL 0
0: jzxh014:2356328:2357230 [1] NCCL INFO P2P Chunksize set to 131072
0: jzxh014:2356330:2357229 [3] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] 0/-1/-1->3->2 [2] -1/-1/-1->3->1 [3] 1/11/-1->3->-1 [4] 2/-1/-1->3->0 [5] 0/-1/-1->3->1 [6] 1/-1/-1->3->0 [7] 2/11/-1->3->-1 [8] -1/-1/-1->3->2 [9] 0/-1/-1->3->2 [10] -1/-1/-1->3->1 [11] 1/-1/-1->3->7 [12] 2/-1/-1->3->0 [13] 0/-1/-1->3->1 [14] 1/-1/-1->3->0 [15] 2/-1/-1->3->7
0: jzxh014:2356330:2357229 [3] NCCL INFO P2P Chunksize set to 131072
1: jzxh015:2965582:2967787 [0] NCCL INFO comm 0x5581ebd8acb0 rank 4 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0
1: jzxh015:2965583:2967788 [1] NCCL INFO comm 0x55fffbc332b0 rank 5 nRanks 16 nNodes 4 localRanks 4 localRank 1 MNNVL 0
1: jzxh015:2965585:2967786 [3] NCCL INFO comm 0x55ee7f078d80 rank 7 nRanks 16 nNodes 4 localRanks 4 localRank 3 MNNVL 0
1: jzxh015:2965584:2967789 [2] NCCL INFO comm 0x564e0a919f70 rank 6 nRanks 16 nNodes 4 localRanks 4 localRank 2 MNNVL 0
0: jzxh014:2356329:2357228 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 0/10/-1->2->-1 [3] -1/-1/-1->2->0 [4] 1/-1/-1->2->3 [5] -1/-1/-1->2->0 [6] 0/10/-1->2->-1 [7] 1/-1/-1->2->3 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 0/-1/-1->2->6 [11] -1/-1/-1->2->0 [12] 1/-1/-1->2->3 [13] -1/-1/-1->2->0 [14] 0/-1/-1->2->6 [15] 1/-1/-1->2->3
0: jzxh014:2356329:2357228 [2] NCCL INFO P2P Chunksize set to 131072
3: jzxh017:51584:52479 [2] NCCL INFO Trees [0] 15/-1/-1->14->13 [1] 15/-1/-1->14->13 [2] 12/-1/-1->14->10 [3] -1/-1/-1->14->12 [4] 13/-1/-1->14->15 [5] -1/-1/-1->14->12 [6] 12/-1/-1->14->10 [7] 13/-1/-1->14->15 [8] 15/-1/-1->14->13 [9] 15/-1/-1->14->13 [10] 12/6/-1->14->-1 [11] -1/-1/-1->14->12 [12] 13/-1/-1->14->15 [13] -1/-1/-1->14->12 [14] 12/6/-1->14->-1 [15] 13/-1/-1->14->15
3: jzxh017:51584:52479 [2] NCCL INFO P2P Chunksize set to 131072
1: jzxh015:2965582:2967787 [0] NCCL INFO Trees [0] 5/-1/-1->4->8 [1] -1/-1/-1->4->7 [2] 5/-1/-1->4->6 [3] 6/-1/-1->4->5 [4] 7/-1/-1->4->8 [5] 6/-1/-1->4->7 [6] 7/-1/-1->4->6 [7] -1/-1/-1->4->5 [8] 5/8/0->4->12 [9] -1/-1/-1->4->7 [10] 5/-1/-1->4->6 [11] 6/-1/-1->4->5 [12] 7/8/0->4->12 [13] 6/-1/-1->4->7 [14] 7/-1/-1->4->6 [15] -1/-1/-1->4->5
1: jzxh015:2965583:2967788 [1] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->9 [2] 7/-1/-1->5->4 [3] 4/-1/-1->5->7 [4] -1/-1/-1->5->6 [5] 7/-1/-1->5->9 [6] -1/-1/-1->5->7 [7] 4/-1/-1->5->6 [8] 6/-1/-1->5->4 [9] 6/9/1->5->13 [10] 7/-1/-1->5->4 [11] 4/-1/-1->5->7 [12] -1/-1/-1->5->6 [13] 7/9/1->5->13 [14] -1/-1/-1->5->7 [15] 4/-1/-1->5->6
1: jzxh015:2965582:2967787 [0] NCCL INFO P2P Chunksize set to 131072
1: jzxh015:2965583:2967788 [1] NCCL INFO P2P Chunksize set to 131072
1: jzxh015:2965584:2967789 [2] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 4/-1/-1->6->10 [3] -1/-1/-1->6->4 [4] 5/-1/-1->6->7 [5] -1/-1/-1->6->4 [6] 4/-1/-1->6->10 [7] 5/-1/-1->6->7 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 4/10/2->6->14 [11] -1/-1/-1->6->4 [12] 5/-1/-1->6->7 [13] -1/-1/-1->6->4 [14] 4/10/2->6->14 [15] 5/-1/-1->6->7
1: jzxh015:2965584:2967789 [2] NCCL INFO P2P Chunksize set to 131072
2: jzxh016:896064:896962 [0] NCCL INFO comm 0x55a50cad5640 rank 8 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0
2: jzxh016:896065:896964 [1] NCCL INFO comm 0x55a46b9ae900 rank 9 nRanks 16 nNodes 4 localRanks 4 localRank 1 MNNVL 0
2: jzxh016:896066:896961 [2] NCCL INFO comm 0x556ee6361e10 rank 10 nRanks 16 nNodes 4 localRanks 4 localRank 2 MNNVL 0
2: jzxh016:896067:896963 [3] NCCL INFO comm 0x5576b8f9bbe0 rank 11 nRanks 16 nNodes 4 localRanks 4 localRank 3 MNNVL 0
2: jzxh016:896064:896962 [0] NCCL INFO Trees [0] 9/4/12->8->0 [1] -1/-1/-1->8->11 [2] 9/-1/-1->8->10 [3] 10/-1/-1->8->9 [4] 11/4/12->8->0 [5] 10/-1/-1->8->11 [6] 11/-1/-1->8->10 [7] -1/-1/-1->8->9 [8] 9/-1/-1->8->4 [9] -1/-1/-1->8->11 [10] 9/-1/-1->8->10 [11] 10/-1/-1->8->9 [12] 11/-1/-1->8->4 [13] 10/-1/-1->8->11 [14] 11/-1/-1->8->10 [15] -1/-1/-1->8->9
2: jzxh016:896064:896962 [0] NCCL INFO P2P Chunksize set to 131072
1: jzxh015:2965585:2967786 [3] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] 4/-1/-1->7->6 [2] -1/-1/-1->7->5 [3] 5/-1/-1->7->11 [4] 6/-1/-1->7->4 [5] 4/-1/-1->7->5 [6] 5/-1/-1->7->4 [7] 6/-1/-1->7->11 [8] -1/-1/-1->7->6 [9] 4/-1/-1->7->6 [10] -1/-1/-1->7->5 [11] 5/11/3->7->15 [12] 6/-1/-1->7->4 [13] 4/-1/-1->7->5 [14] 5/-1/-1->7->4 [15] 6/11/3->7->15
1: jzxh015:2965585:2967786 [3] NCCL INFO P2P Chunksize set to 131072
3: jzxh017:51583:52476 [1] NCCL INFO comm 0x558abda268d0 rank 13 nRanks 16 nNodes 4 localRanks 4 localRank 1 MNNVL 0
3: jzxh017:51583:52476 [1] NCCL INFO Trees [0] 14/-1/-1->13->12 [1] 14/-1/-1->13->9 [2] 15/-1/-1->13->12 [3] 12/-1/-1->13->15 [4] -1/-1/-1->13->14 [5] 15/-1/-1->13->9 [6] -1/-1/-1->13->15 [7] 12/-1/-1->13->14 [8] 14/-1/-1->13->12 [9] 14/5/-1->13->-1 [10] 15/-1/-1->13->12 [11] 12/-1/-1->13->15 [12] -1/-1/-1->13->14 [13] 15/5/-1->13->-1 [14] -1/-1/-1->13->15 [15] 12/-1/-1->13->14
3: jzxh017:51583:52476 [1] NCCL INFO P2P Chunksize set to 131072
3: jzxh017:51582:52477 [0] NCCL INFO comm 0x559e3dba4770 rank 12 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0
3: jzxh017:51582:52477 [0] NCCL INFO Trees [0] 13/-1/-1->12->8 [1] -1/-1/-1->12->15 [2] 13/-1/-1->12->14 [3] 14/-1/-1->12->13 [4] 15/-1/-1->12->8 [5] 14/-1/-1->12->15 [6] 15/-1/-1->12->14 [7] -1/-1/-1->12->13 [8] 13/4/-1->12->-1 [9] -1/-1/-1->12->15 [10] 13/-1/-1->12->14 [11] 14/-1/-1->12->13 [12] 15/4/-1->12->-1 [13] 14/-1/-1->12->15 [14] 15/-1/-1->12->14 [15] -1/-1/-1->12->13
3: jzxh017:51582:52477 [0] NCCL INFO P2P Chunksize set to 131072
2: jzxh016:896065:896964 [1] NCCL INFO Trees [0] 10/-1/-1->9->8 [1] 10/5/13->9->1 [2] 11/-1/-1->9->8 [3] 8/-1/-1->9->11 [4] -1/-1/-1->9->10 [5] 11/5/13->9->1 [6] -1/-1/-1->9->11 [7] 8/-1/-1->9->10 [8] 10/-1/-1->9->8 [9] 10/-1/-1->9->5 [10] 11/-1/-1->9->8 [11] 8/-1/-1->9->11 [12] -1/-1/-1->9->10 [13] 11/-1/-1->9->5 [14] -1/-1/-1->9->11 [15] 8/-1/-1->9->10
2: jzxh016:896065:896964 [1] NCCL INFO P2P Chunksize set to 131072
2: jzxh016:896066:896961 [2] NCCL INFO Trees [0] 11/-1/-1->10->9 [1] 11/-1/-1->10->9 [2] 8/6/14->10->2 [3] -1/-1/-1->10->8 [4] 9/-1/-1->10->11 [5] -1/-1/-1->10->8 [6] 8/6/14->10->2 [7] 9/-1/-1->10->11 [8] 11/-1/-1->10->9 [9] 11/-1/-1->10->9 [10] 8/-1/-1->10->6 [11] -1/-1/-1->10->8 [12] 9/-1/-1->10->11 [13] -1/-1/-1->10->8 [14] 8/-1/-1->10->6 [15] 9/-1/-1->10->11
2: jzxh016:896066:896961 [2] NCCL INFO P2P Chunksize set to 131072
2: jzxh016:896067:896963 [3] NCCL INFO Trees [0] -1/-1/-1->11->10 [1] 8/-1/-1->11->10 [2] -1/-1/-1->11->9 [3] 9/7/15->11->3 [4] 10/-1/-1->11->8 [5] 8/-1/-1->11->9 [6] 9/-1/-1->11->8 [7] 10/7/15->11->3 [8] -1/-1/-1->11->10 [9] 8/-1/-1->11->10 [10] -1/-1/-1->11->9 [11] 9/-1/-1->11->7 [12] 10/-1/-1->11->8 [13] 8/-1/-1->11->9 [14] 9/-1/-1->11->8 [15] 10/-1/-1->11->7
2: jzxh016:896067:896963 [3] NCCL INFO P2P Chunksize set to 131072
2: jzxh016:896066:896961 [2] NCCL INFO Channel 00/0 : 10[2] -> 11[3] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 00/0 : 5[1] -> 6[2] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 00/0 : 13[1] -> 14[2] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 04/0 : 10[2] -> 11[3] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 00/0 : 6[2] -> 7[3] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 04/0 : 1[1] -> 2[2] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 03/0 : 13[1] -> 14[2] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 08/0 : 10[2] -> 11[3] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 07/0 : 1[1] -> 2[2] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 00/0 : 14[2] -> 15[3] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 04/0 : 13[1] -> 14[2] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 04/0 : 6[2] -> 7[3] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 07/0 : 13[1] -> 14[2] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 03/0 : 5[1] -> 6[2] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 12/0 : 10[2] -> 11[3] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 08/0 : 1[1] -> 2[2] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 08/0 : 13[1] -> 14[2] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 08/0 : 6[2] -> 7[3] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 04/0 : 5[1] -> 6[2] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 11/0 : 1[1] -> 2[2] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 04/0 : 14[2] -> 15[3] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 12/0 : 1[1] -> 2[2] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 00/0 : 9[1] -> 10[2] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 12/0 : 6[2] -> 7[3] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 07/0 : 5[1] -> 6[2] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 15/0 : 1[1] -> 2[2] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 08/0 : 5[1] -> 6[2] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 11/0 : 13[1] -> 14[2] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 03/0 : 9[1] -> 10[2] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 12/0 : 13[1] -> 14[2] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 11/0 : 5[1] -> 6[2] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 08/0 : 14[2] -> 15[3] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 04/0 : 9[1] -> 10[2] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 12/0 : 14[2] -> 15[3] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 07/0 : 9[1] -> 10[2] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 12/0 : 5[1] -> 6[2] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 15/0 : 13[1] -> 14[2] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 08/0 : 9[1] -> 10[2] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 15/0 : 5[1] -> 6[2] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 11/0 : 9[1] -> 10[2] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 12/0 : 9[1] -> 10[2] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 15/0 : 9[1] -> 10[2] via P2P/CUMEM
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 00/0 : 3[3] -> 4[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 00/0 : 7[3] -> 8[0] [send] via NET/IB/0(4)/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 04/0 : 3[3] -> 4[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 04/0 : 7[3] -> 8[0] [send] via NET/IB/0(4)/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 08/0 : 3[3] -> 4[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 08/0 : 7[3] -> 8[0] [send] via NET/IB/0(4)/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 12/0 : 3[3] -> 4[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 00/0 : 4[0] -> 5[1] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 12/0 : 7[3] -> 8[0] [send] via NET/IB/0(4)/GDRDMA
2: jzxh016:896064:896962 [0] NCCL INFO Channel 00/0 : 7[3] -> 8[0] [receive] via NET/IB/0/GDRDMA
3: jzxh017:51582:52477 [0] NCCL INFO Channel 00/0 : 11[3] -> 12[0] [receive] via NET/IB/0/GDRDMA
3: jzxh017:51585:52478 [3] NCCL INFO Channel 00/0 : 15[3] -> 0[0] [send] via NET/IB/0(12)/GDRDMA
2: jzxh016:896067:896963 [3] NCCL INFO Channel 00/0 : 11[3] -> 12[0] [send] via NET/IB/0(8)/GDRDMA
2: jzxh016:896064:896962 [0] NCCL INFO Channel 04/0 : 7[3] -> 8[0] [receive] via NET/IB/0/GDRDMA
2: jzxh016:896067:896963 [3] NCCL INFO Channel 04/0 : 11[3] -> 12[0] [send] via NET/IB/0(8)/GDRDMA
2: jzxh016:896064:896962 [0] NCCL INFO Channel 08/0 : 7[3] -> 8[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 03/0 : 4[0] -> 5[1] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 00/0 : 15[3] -> 0[0] [receive] via NET/IB/0/GDRDMA
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[0] [send] via NET/IB/0(0)/GDRDMA
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 04/0 : 15[3] -> 0[0] [receive] via NET/IB/0/GDRDMA
3: jzxh017:51582:52477 [0] NCCL INFO Channel 04/0 : 11[3] -> 12[0] [receive] via NET/IB/0/GDRDMA
2: jzxh016:896067:896963 [3] NCCL INFO Channel 08/0 : 11[3] -> 12[0] [send] via NET/IB/0(8)/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 04/0 : 4[0] -> 5[1] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 12/0 : 7[3] -> 8[0] [receive] via NET/IB/0/GDRDMA
2: jzxh016:896064:896962 [0] NCCL INFO Channel 00/0 : 8[0] -> 9[1] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 04/0 : 3[3] -> 4[0] [send] via NET/IB/0(0)/GDRDMA
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 08/0 : 15[3] -> 0[0] [receive] via NET/IB/0/GDRDMA
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 08/0 : 3[3] -> 4[0] [send] via NET/IB/0(0)/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 07/0 : 4[0] -> 5[1] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 12/0 : 15[3] -> 0[0] [receive] via NET/IB/0/GDRDMA
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 04/0 : 15[3] -> 0[0] [send] via NET/IB/0(12)/GDRDMA
3: jzxh017:51582:52477 [0] NCCL INFO Channel 08/0 : 11[3] -> 12[0] [receive] via NET/IB/0/GDRDMA
3: jzxh017:51585:52478 [3] NCCL INFO Channel 08/0 : 15[3] -> 0[0] [send] via NET/IB/0(12)/GDRDMA
2: jzxh016:896067:896963 [3] NCCL INFO Channel 12/0 : 11[3] -> 12[0] [send] via NET/IB/0(8)/GDRDMA
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 12/0 : 3[3] -> 4[0] [send] via NET/IB/0(0)/GDRDMA
3: jzxh017:51582:52477 [0] NCCL INFO Channel 12/0 : 11[3] -> 12[0] [receive] via NET/IB/0/GDRDMA
3: jzxh017:51582:52477 [0] NCCL INFO Channel 00/0 : 12[0] -> 13[1] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 12/0 : 15[3] -> 0[0] [send] via NET/IB/0(12)/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 08/0 : 4[0] -> 5[1] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 03/0 : 8[0] -> 9[1] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 03/0 : 12[0] -> 13[1] via P2P/CUMEM
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 11/0 : 4[0] -> 5[1] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 04/0 : 12[0] -> 13[1] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 04/0 : 8[0] -> 9[1] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 07/0 : 8[0] -> 9[1] via P2P/CUMEM
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 12/0 : 4[0] -> 5[1] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 07/0 : 12[0] -> 13[1] via P2P/CUMEM
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 15/0 : 4[0] -> 5[1] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 08/0 : 8[0] -> 9[1] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 11/0 : 8[0] -> 9[1] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 08/0 : 12[0] -> 13[1] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 12/0 : 8[0] -> 9[1] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 11/0 : 12[0] -> 13[1] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 12/0 : 12[0] -> 13[1] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 15/0 : 12[0] -> 13[1] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 15/0 : 8[0] -> 9[1] via P2P/CUMEM
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 01/0 : 4[0] -> 7[3] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/CUMEM
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 02/0 : 4[0] -> 7[3] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 01/0 : 12[0] -> 15[3] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 01/0 : 8[0] -> 11[3] via P2P/CUMEM
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 05/0 : 4[0] -> 7[3] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 02/0 : 8[0] -> 11[3] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 02/0 : 12[0] -> 15[3] via P2P/CUMEM
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 06/0 : 4[0] -> 7[3] via P2P/CUMEM
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 09/0 : 4[0] -> 7[3] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 05/0 : 12[0] -> 15[3] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 05/0 : 8[0] -> 11[3] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 06/0 : 12[0] -> 15[3] via P2P/CUMEM
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 10/0 : 4[0] -> 7[3] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 01/0 : 0[0] -> 3[3] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 06/0 : 8[0] -> 11[3] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 09/0 : 12[0] -> 15[3] via P2P/CUMEM
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 13/0 : 4[0] -> 7[3] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 02/0 : 0[0] -> 3[3] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 10/0 : 12[0] -> 15[3] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 09/0 : 8[0] -> 11[3] via P2P/CUMEM
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 14/0 : 4[0] -> 7[3] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 05/0 : 0[0] -> 3[3] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 10/0 : 8[0] -> 11[3] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 06/0 : 0[0] -> 3[3] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 13/0 : 12[0] -> 15[3] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 14/0 : 12[0] -> 15[3] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 09/0 : 0[0] -> 3[3] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 13/0 : 8[0] -> 11[3] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 10/0 : 0[0] -> 3[3] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 14/0 : 8[0] -> 11[3] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 13/0 : 0[0] -> 3[3] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 14/0 : 0[0] -> 3[3] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 02/0 : 3[3] -> 6[2] [receive] via NET/IB/2/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 02/0 : 7[3] -> 10[2] [send] via NET/IB/2(6)/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 02/0 : 7[3] -> 10[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896067:896963 [3] NCCL INFO Channel 02/0 : 11[3] -> 14[2] [send] via NET/IB/2(10)/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 06/0 : 3[3] -> 6[2] [receive] via NET/IB/2/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 06/0 : 7[3] -> 10[2] [send] via NET/IB/2(6)/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 10/0 : 3[3] -> 6[2] [receive] via NET/IB/2/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 10/0 : 7[3] -> 10[2] [send] via NET/IB/2(6)/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 14/0 : 3[3] -> 6[2] [receive] via NET/IB/2/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 14/0 : 7[3] -> 10[2] [send] via NET/IB/2(6)/GDRDMA
3: jzxh017:51584:52479 [2] NCCL INFO Channel 02/0 : 11[3] -> 14[2] [receive] via NET/IB/2/GDRDMA
3: jzxh017:51585:52478 [3] NCCL INFO Channel 02/0 : 15[3] -> 2[2] [send] via NET/IB/2(14)/GDRDMA
3: jzxh017:51584:52479 [2] NCCL INFO Channel 06/0 : 11[3] -> 14[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 06/0 : 7[3] -> 10[2] [receive] via NET/IB/2/GDRDMA
3: jzxh017:51585:52478 [3] NCCL INFO Channel 06/0 : 15[3] -> 2[2] [send] via NET/IB/2(14)/GDRDMA
2: jzxh016:896067:896963 [3] NCCL INFO Channel 06/0 : 11[3] -> 14[2] [send] via NET/IB/2(10)/GDRDMA
3: jzxh017:51584:52479 [2] NCCL INFO Channel 10/0 : 11[3] -> 14[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 10/0 : 7[3] -> 10[2] [receive] via NET/IB/2/GDRDMA
3: jzxh017:51585:52478 [3] NCCL INFO Channel 10/0 : 15[3] -> 2[2] [send] via NET/IB/2(14)/GDRDMA
2: jzxh016:896067:896963 [3] NCCL INFO Channel 10/0 : 11[3] -> 14[2] [send] via NET/IB/2(10)/GDRDMA
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 02/0 : 15[3] -> 2[2] [receive] via NET/IB/2/GDRDMA
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 02/0 : 3[3] -> 6[2] [send] via NET/IB/2(2)/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 14/0 : 7[3] -> 10[2] [receive] via NET/IB/2/GDRDMA
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 06/0 : 15[3] -> 2[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896067:896963 [3] NCCL INFO Channel 14/0 : 11[3] -> 14[2] [send] via NET/IB/2(10)/GDRDMA
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 06/0 : 3[3] -> 6[2] [send] via NET/IB/2(2)/GDRDMA
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 10/0 : 15[3] -> 2[2] [receive] via NET/IB/2/GDRDMA
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 10/0 : 3[3] -> 6[2] [send] via NET/IB/2(2)/GDRDMA
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 14/0 : 15[3] -> 2[2] [receive] via NET/IB/2/GDRDMA
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 14/0 : 3[3] -> 6[2] [send] via NET/IB/2(2)/GDRDMA
3: jzxh017:51584:52479 [2] NCCL INFO Channel 14/0 : 11[3] -> 14[2] [receive] via NET/IB/2/GDRDMA
3: jzxh017:51585:52478 [3] NCCL INFO Channel 14/0 : 15[3] -> 2[2] [send] via NET/IB/2(14)/GDRDMA
2: jzxh016:896065:896964 [1] NCCL INFO Channel 01/0 : 6[2] -> 9[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 01/0 : 10[2] -> 13[1] [send] via NET/IB/1(9)/GDRDMA
2: jzxh016:896065:896964 [1] NCCL INFO Channel 05/0 : 6[2] -> 9[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 05/0 : 10[2] -> 13[1] [send] via NET/IB/1(9)/GDRDMA
2: jzxh016:896065:896964 [1] NCCL INFO Channel 09/0 : 6[2] -> 9[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 09/0 : 10[2] -> 13[1] [send] via NET/IB/1(9)/GDRDMA
2: jzxh016:896065:896964 [1] NCCL INFO Channel 13/0 : 6[2] -> 9[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 13/0 : 10[2] -> 13[1] [send] via NET/IB/1(9)/GDRDMA
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 01/0 : 2[2] -> 5[1] [receive] via NET/IB/1/GDRDMA
3: jzxh017:51583:52476 [1] NCCL INFO Channel 01/0 : 10[2] -> 13[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 01/0 : 6[2] -> 9[1] [send] via NET/IB/1(5)/GDRDMA
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 05/0 : 2[2] -> 5[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 05/0 : 6[2] -> 9[1] [send] via NET/IB/1(5)/GDRDMA
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 09/0 : 2[2] -> 5[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 09/0 : 6[2] -> 9[1] [send] via NET/IB/1(5)/GDRDMA
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 13/0 : 2[2] -> 5[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 13/0 : 6[2] -> 9[1] [send] via NET/IB/1(5)/GDRDMA
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 01/0 : 14[2] -> 1[1] [receive] via NET/IB/1/GDRDMA
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 01/0 : 2[2] -> 5[1] [send] via NET/IB/1(1)/GDRDMA
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 05/0 : 14[2] -> 1[1] [receive] via NET/IB/1/GDRDMA
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 05/0 : 2[2] -> 5[1] [send] via NET/IB/1(1)/GDRDMA
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 09/0 : 14[2] -> 1[1] [receive] via NET/IB/1/GDRDMA
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 09/0 : 2[2] -> 5[1] [send] via NET/IB/1(1)/GDRDMA
3: jzxh017:51584:52479 [2] NCCL INFO Channel 01/0 : 14[2] -> 1[1] [send] via NET/IB/1(13)/GDRDMA
3: jzxh017:51583:52476 [1] NCCL INFO Channel 05/0 : 10[2] -> 13[1] [receive] via NET/IB/1/GDRDMA
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 13/0 : 14[2] -> 1[1] [receive] via NET/IB/1/GDRDMA
3: jzxh017:51584:52479 [2] NCCL INFO Channel 05/0 : 14[2] -> 1[1] [send] via NET/IB/1(13)/GDRDMA
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 13/0 : 2[2] -> 5[1] [send] via NET/IB/1(1)/GDRDMA
3: jzxh017:51583:52476 [1] NCCL INFO Channel 09/0 : 10[2] -> 13[1] [receive] via NET/IB/1/GDRDMA
3: jzxh017:51584:52479 [2] NCCL INFO Channel 09/0 : 14[2] -> 1[1] [send] via NET/IB/1(13)/GDRDMA
3: jzxh017:51583:52476 [1] NCCL INFO Channel 13/0 : 10[2] -> 13[1] [receive] via NET/IB/1/GDRDMA
3: jzxh017:51584:52479 [2] NCCL INFO Channel 13/0 : 14[2] -> 1[1] [send] via NET/IB/1(13)/GDRDMA
2: jzxh016:896065:896964 [1] NCCL INFO Channel 01/0 : 9[1] -> 8[0] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 02/0 : 9[1] -> 8[0] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 01/0 : 5[1] -> 4[0] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 05/0 : 9[1] -> 8[0] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 06/0 : 9[1] -> 8[0] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 02/0 : 5[1] -> 4[0] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 01/0 : 13[1] -> 12[0] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 09/0 : 9[1] -> 8[0] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 05/0 : 5[1] -> 4[0] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 06/0 : 5[1] -> 4[0] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 02/0 : 13[1] -> 12[0] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 10/0 : 9[1] -> 8[0] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 05/0 : 13[1] -> 12[0] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 09/0 : 5[1] -> 4[0] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 13/0 : 9[1] -> 8[0] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 14/0 : 9[1] -> 8[0] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 06/0 : 13[1] -> 12[0] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 10/0 : 5[1] -> 4[0] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 09/0 : 13[1] -> 12[0] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 13/0 : 5[1] -> 4[0] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 10/0 : 13[1] -> 12[0] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 14/0 : 5[1] -> 4[0] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 13/0 : 13[1] -> 12[0] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 14/0 : 13[1] -> 12[0] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 03/0 : 2[2] -> 7[3] [receive] via NET/IB/3/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 03/0 : 6[2] -> 11[3] [send] via NET/IB/3(7)/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 07/0 : 2[2] -> 7[3] [receive] via NET/IB/3/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 07/0 : 6[2] -> 11[3] [send] via NET/IB/3(7)/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 11/0 : 2[2] -> 7[3] [receive] via NET/IB/3/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 11/0 : 6[2] -> 11[3] [send] via NET/IB/3(7)/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 15/0 : 2[2] -> 7[3] [receive] via NET/IB/3/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 15/0 : 6[2] -> 11[3] [send] via NET/IB/3(7)/GDRDMA
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 03/0 : 14[2] -> 3[3] [receive] via NET/IB/3/GDRDMA
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 03/0 : 2[2] -> 7[3] [send] via NET/IB/3(3)/GDRDMA
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 07/0 : 14[2] -> 3[3] [receive] via NET/IB/3/GDRDMA
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 07/0 : 2[2] -> 7[3] [send] via NET/IB/3(3)/GDRDMA
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 11/0 : 14[2] -> 3[3] [receive] via NET/IB/3/GDRDMA
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 11/0 : 2[2] -> 7[3] [send] via NET/IB/3(3)/GDRDMA
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 15/0 : 14[2] -> 3[3] [receive] via NET/IB/3/GDRDMA
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 15/0 : 2[2] -> 7[3] [send] via NET/IB/3(3)/GDRDMA
3: jzxh017:51585:52478 [3] NCCL INFO Channel 03/0 : 10[2] -> 15[3] [receive] via NET/IB/3/GDRDMA
3: jzxh017:51584:52479 [2] NCCL INFO Channel 03/0 : 14[2] -> 3[3] [send] via NET/IB/3(15)/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 03/0 : 7[3] -> 4[0] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 07/0 : 7[3] -> 4[0] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 03/0 : 3[3] -> 0[0] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 07/0 : 10[2] -> 15[3] [receive] via NET/IB/3/GDRDMA
3: jzxh017:51584:52479 [2] NCCL INFO Channel 07/0 : 14[2] -> 3[3] [send] via NET/IB/3(15)/GDRDMA
3: jzxh017:51585:52478 [3] NCCL INFO Channel 11/0 : 10[2] -> 15[3] [receive] via NET/IB/3/GDRDMA
3: jzxh017:51584:52479 [2] NCCL INFO Channel 11/0 : 14[2] -> 3[3] [send] via NET/IB/3(15)/GDRDMA
3: jzxh017:51585:52478 [3] NCCL INFO Channel 15/0 : 10[2] -> 15[3] [receive] via NET/IB/3/GDRDMA
3: jzxh017:51584:52479 [2] NCCL INFO Channel 15/0 : 14[2] -> 3[3] [send] via NET/IB/3(15)/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 11/0 : 7[3] -> 4[0] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 07/0 : 3[3] -> 0[0] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 02/0 : 2[2] -> 1[1] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 15/0 : 7[3] -> 4[0] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 11/0 : 3[3] -> 0[0] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 06/0 : 2[2] -> 1[1] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 15/0 : 3[3] -> 0[0] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 03/0 : 6[2] -> 11[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 03/0 : 10[2] -> 15[3] [send] via NET/IB/3(11)/GDRDMA
2: jzxh016:896067:896963 [3] NCCL INFO Channel 07/0 : 6[2] -> 11[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 07/0 : 10[2] -> 15[3] [send] via NET/IB/3(11)/GDRDMA
2: jzxh016:896067:896963 [3] NCCL INFO Channel 11/0 : 6[2] -> 11[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 11/0 : 10[2] -> 15[3] [send] via NET/IB/3(11)/GDRDMA
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 10/0 : 2[2] -> 1[1] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 01/0 : 7[3] -> 6[2] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 14/0 : 2[2] -> 1[1] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 03/0 : 15[3] -> 12[0] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 15/0 : 6[2] -> 11[3] [receive] via NET/IB/3/GDRDMA
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 15/0 : 10[2] -> 15[3] [send] via NET/IB/3(11)/GDRDMA
2: jzxh016:896067:896963 [3] NCCL INFO Channel 03/0 : 11[3] -> 8[0] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 05/0 : 7[3] -> 6[2] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 02/0 : 14[2] -> 13[1] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 06/0 : 14[2] -> 13[1] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 07/0 : 11[3] -> 8[0] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 09/0 : 7[3] -> 6[2] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 07/0 : 15[3] -> 12[0] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 10/0 : 14[2] -> 13[1] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 11/0 : 11[3] -> 8[0] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 11/0 : 15[3] -> 12[0] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 15/0 : 11[3] -> 8[0] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 14/0 : 14[2] -> 13[1] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 15/0 : 15[3] -> 12[0] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 13/0 : 7[3] -> 6[2] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 02/0 : 10[2] -> 9[1] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 01/0 : 15[3] -> 14[2] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 01/0 : 11[3] -> 10[2] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 06/0 : 10[2] -> 9[1] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 05/0 : 15[3] -> 14[2] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 05/0 : 11[3] -> 10[2] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 10/0 : 10[2] -> 9[1] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 02/0 : 6[2] -> 5[1] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 09/0 : 15[3] -> 14[2] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 14/0 : 10[2] -> 9[1] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 06/0 : 6[2] -> 5[1] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 13/0 : 15[3] -> 14[2] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 09/0 : 11[3] -> 10[2] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 10/0 : 6[2] -> 5[1] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 13/0 : 11[3] -> 10[2] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 14/0 : 6[2] -> 5[1] via P2P/CUMEM
1: jzxh015:2965582:2967787 [0] NCCL INFO Connected all rings
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 02/0 : 4[0] -> 5[1] via P2P/CUMEM
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 10/0 : 4[0] -> 5[1] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Connected all rings
0: jzxh014:2356330:2357229 [3] NCCL INFO Connected all rings
0: jzxh014:2356327:2357227 [0] NCCL INFO Connected all rings
0: jzxh014:2356328:2357230 [1] NCCL INFO Connected all rings
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Connected all rings
2: jzxh016:896066:896961 [2] NCCL INFO Connected all rings
2: jzxh016:896067:896963 [3] NCCL INFO Connected all rings
2: jzxh016:896064:896962 [0] NCCL INFO Connected all rings
1: jzxh015:2965585:2967786 [3] NCCL INFO Connected all rings
1: jzxh015:2965584:2967789 [2] NCCL INFO Connected all rings
3: jzxh017:51583:52476 [1] NCCL INFO Connected all rings
3: jzxh017:51582:52477 [0] NCCL INFO Connected all rings
3: jzxh017:51584:52479 [2] NCCL INFO Connected all rings
3: jzxh017:51585:52478 [3] NCCL INFO Connected all rings
3: jzxh017:51582:52477 [0] NCCL INFO Channel 02/0 : 12[0] -> 13[1] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Connected all rings
2: jzxh016:896064:896962 [0] NCCL INFO Channel 02/0 : 8[0] -> 9[1] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 10/0 : 12[0] -> 13[1] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 10/0 : 8[0] -> 9[1] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 01/0 : 6[2] -> 7[3] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 09/0 : 1[1] -> 2[2] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 01/0 : 14[2] -> 15[3] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 01/0 : 10[2] -> 11[3] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 01/0 : 5[1] -> 6[2] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 01/0 : 13[1] -> 14[2] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 07/0 : 6[2] -> 7[3] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 02/0 : 0[0] -> 2[2] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 07/0 : 10[2] -> 11[3] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 07/0 : 14[2] -> 15[3] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 09/0 : 13[1] -> 14[2] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 09/0 : 5[1] -> 6[2] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 09/0 : 6[2] -> 7[3] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 01/0 : 9[1] -> 10[2] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 03/0 : 0[0] -> 2[2] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 09/0 : 14[2] -> 15[3] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 09/0 : 10[2] -> 11[3] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 15/0 : 6[2] -> 7[3] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 09/0 : 9[1] -> 10[2] via P2P/CUMEM
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 02/0 : 4[0] -> 6[2] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 02/0 : 12[0] -> 14[2] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 15/0 : 10[2] -> 11[3] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 15/0 : 14[2] -> 15[3] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 05/0 : 0[0] -> 2[2] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 03/0 : 12[0] -> 14[2] via P2P/CUMEM
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 03/0 : 4[0] -> 6[2] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 06/0 : 0[0] -> 2[2] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 02/0 : 1[1] -> 3[3] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 05/0 : 12[0] -> 14[2] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 02/0 : 5[1] -> 7[3] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 10/0 : 0[0] -> 2[2] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 03/0 : 1[1] -> 3[3] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 02/0 : 13[1] -> 15[3] via P2P/CUMEM
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 05/0 : 4[0] -> 6[2] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 03/0 : 5[1] -> 7[3] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 11/0 : 0[0] -> 2[2] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 05/0 : 1[1] -> 3[3] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 06/0 : 12[0] -> 14[2] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 02/0 : 8[0] -> 10[2] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 02/0 : 9[1] -> 11[3] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 05/0 : 5[1] -> 7[3] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 03/0 : 13[1] -> 15[3] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 13/0 : 0[0] -> 2[2] via P2P/CUMEM
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 06/0 : 4[0] -> 6[2] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 06/0 : 1[1] -> 3[3] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 03/0 : 9[1] -> 11[3] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 14/0 : 0[0] -> 2[2] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 10/0 : 12[0] -> 14[2] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 06/0 : 5[1] -> 7[3] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 10/0 : 1[1] -> 3[3] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 05/0 : 9[1] -> 11[3] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 05/0 : 13[1] -> 15[3] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 03/0 : 8[0] -> 10[2] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 11/0 : 1[1] -> 3[3] via P2P/CUMEM
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 10/0 : 4[0] -> 6[2] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 11/0 : 12[0] -> 14[2] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 13/0 : 1[1] -> 3[3] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 06/0 : 13[1] -> 15[3] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 06/0 : 9[1] -> 11[3] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 10/0 : 5[1] -> 7[3] via P2P/CUMEM
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 11/0 : 4[0] -> 6[2] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 05/0 : 8[0] -> 10[2] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 14/0 : 1[1] -> 3[3] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 10/0 : 13[1] -> 15[3] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 11/0 : 5[1] -> 7[3] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 10/0 : 9[1] -> 11[3] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 06/0 : 8[0] -> 10[2] via P2P/CUMEM
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 13/0 : 4[0] -> 6[2] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 11/0 : 13[1] -> 15[3] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 13/0 : 13[1] -> 15[3] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 10/0 : 8[0] -> 10[2] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 13/0 : 5[1] -> 7[3] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 11/0 : 9[1] -> 11[3] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 13/0 : 12[0] -> 14[2] via P2P/CUMEM
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 14/0 : 4[0] -> 6[2] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 14/0 : 13[1] -> 15[3] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 13/0 : 9[1] -> 11[3] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 14/0 : 5[1] -> 7[3] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 14/0 : 12[0] -> 14[2] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 11/0 : 8[0] -> 10[2] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 04/0 : 12[0] -> 15[3] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 14/0 : 9[1] -> 11[3] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 09/0 : 1[1] -> 5[1] [send] via NET/IB/1/GDRDMA
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 04/0 : 0[0] -> 3[3] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 13/0 : 1[1] -> 5[1] [send] via NET/IB/1/GDRDMA
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 10/0 : 2[2] -> 6[2] [send] via NET/IB/2/GDRDMA
3: jzxh017:51584:52479 [2] NCCL INFO Channel 02/0 : 10[2] -> 14[2] [receive] via NET/IB/2/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 04/0 : 4[0] -> 7[3] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 14/0 : 2[2] -> 6[2] [send] via NET/IB/2/GDRDMA
2: jzxh016:896064:896962 [0] NCCL INFO Channel 13/0 : 8[0] -> 10[2] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 12/0 : 0[0] -> 3[3] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 10/0 : 2[2] -> 6[2] [receive] via NET/IB/2/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 14/0 : 2[2] -> 6[2] [receive] via NET/IB/2/GDRDMA
3: jzxh017:51584:52479 [2] NCCL INFO Channel 06/0 : 10[2] -> 14[2] [receive] via NET/IB/2/GDRDMA
3: jzxh017:51582:52477 [0] NCCL INFO Channel 12/0 : 12[0] -> 15[3] via P2P/CUMEM
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 12/0 : 4[0] -> 7[3] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 02/0 : 6[2] -> 10[2] [send] via NET/IB/2/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 06/0 : 6[2] -> 10[2] [send] via NET/IB/2/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 10/0 : 6[2] -> 10[2] [send] via NET/IB/2/GDRDMA
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 09/0 : 1[1] -> 5[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896064:896962 [0] NCCL INFO Channel 14/0 : 8[0] -> 10[2] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 11/0 : 3[3] -> 7[3] [send] via NET/IB/3/GDRDMA
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 08/0 : 0[0] -> 4[0] [send] via NET/IB/0/GDRDMA
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 15/0 : 3[3] -> 7[3] [send] via NET/IB/3/GDRDMA
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 12/0 : 0[0] -> 4[0] [send] via NET/IB/0/GDRDMA
2: jzxh016:896065:896964 [1] NCCL INFO Channel 01/0 : 5[1] -> 9[1] [receive] via NET/IB/1/GDRDMA
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 02/0 : 10[2] -> 2[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896065:896964 [1] NCCL INFO Channel 05/0 : 5[1] -> 9[1] [receive] via NET/IB/1/GDRDMA
3: jzxh017:51583:52476 [1] NCCL INFO Channel 01/0 : 9[1] -> 13[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896065:896964 [1] NCCL INFO Channel 09/0 : 5[1] -> 9[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 14/0 : 6[2] -> 10[2] [send] via NET/IB/2/GDRDMA
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 06/0 : 10[2] -> 2[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896064:896962 [0] NCCL INFO Channel 04/0 : 8[0] -> 11[3] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 13/0 : 1[1] -> 5[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 01/0 : 5[1] -> 9[1] [send] via NET/IB/1/GDRDMA
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 05/0 : 5[1] -> 9[1] [send] via NET/IB/1/GDRDMA
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 09/0 : 5[1] -> 9[1] [send] via NET/IB/1/GDRDMA
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 02/0 : 2[2] -> 10[2] [send] via NET/IB/2/GDRDMA
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 06/0 : 2[2] -> 10[2] [send] via NET/IB/2/GDRDMA
2: jzxh016:896065:896964 [1] NCCL INFO Channel 13/0 : 5[1] -> 9[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 02/0 : 6[2] -> 10[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896065:896964 [1] NCCL INFO Channel 01/0 : 9[1] -> 13[1] [send] via NET/IB/1/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 06/0 : 6[2] -> 10[2] [receive] via NET/IB/2/GDRDMA
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 01/0 : 9[1] -> 1[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896065:896964 [1] NCCL INFO Channel 05/0 : 9[1] -> 13[1] [send] via NET/IB/1/GDRDMA
2: jzxh016:896064:896962 [0] NCCL INFO Channel 12/0 : 8[0] -> 11[3] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 05/0 : 9[1] -> 13[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 10/0 : 6[2] -> 10[2] [receive] via NET/IB/2/GDRDMA
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 13/0 : 5[1] -> 9[1] [send] via NET/IB/1/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 11/0 : 3[3] -> 7[3] [receive] via NET/IB/3/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 08/0 : 0[0] -> 4[0] [receive] via NET/IB/0/GDRDMA
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 05/0 : 9[1] -> 1[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 12/0 : 0[0] -> 4[0] [receive] via NET/IB/0/GDRDMA
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 01/0 : 1[1] -> 9[1] [send] via NET/IB/1/GDRDMA
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 05/0 : 1[1] -> 9[1] [send] via NET/IB/1/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 15/0 : 3[3] -> 7[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 14/0 : 6[2] -> 10[2] [receive] via NET/IB/2/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 00/0 : 4[0] -> 8[0] [send] via NET/IB/0/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 02/0 : 10[2] -> 14[2] [send] via NET/IB/2/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 03/0 : 7[3] -> 11[3] [send] via NET/IB/3/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 06/0 : 10[2] -> 14[2] [send] via NET/IB/2/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 04/0 : 4[0] -> 8[0] [send] via NET/IB/0/GDRDMA
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 09/0 : 13[1] -> 5[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 07/0 : 7[3] -> 11[3] [send] via NET/IB/3/GDRDMA
3: jzxh017:51582:52477 [0] NCCL INFO Channel 00/0 : 8[0] -> 12[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 08/0 : 4[0] -> 8[0] [send] via NET/IB/0/GDRDMA
3: jzxh017:51583:52476 [1] NCCL INFO Channel 09/0 : 5[1] -> 13[1] [receive] via NET/IB/1/GDRDMA
3: jzxh017:51585:52478 [3] NCCL INFO Channel 03/0 : 11[3] -> 15[3] [receive] via NET/IB/3/GDRDMA
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 00/0 : 8[0] -> 0[0] [receive] via NET/IB/0/GDRDMA
3: jzxh017:51582:52477 [0] NCCL INFO Channel 04/0 : 8[0] -> 12[0] [receive] via NET/IB/0/GDRDMA
2: jzxh016:896065:896964 [1] NCCL INFO Channel 01/0 : 1[1] -> 9[1] [receive] via NET/IB/1/GDRDMA
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 04/0 : 8[0] -> 0[0] [receive] via NET/IB/0/GDRDMA
3: jzxh017:51583:52476 [1] NCCL INFO Channel 13/0 : 5[1] -> 13[1] [receive] via NET/IB/1/GDRDMA
3: jzxh017:51585:52478 [3] NCCL INFO Channel 07/0 : 11[3] -> 15[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896067:896963 [3] NCCL INFO Channel 03/0 : 7[3] -> 11[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896064:896962 [0] NCCL INFO Channel 00/0 : 4[0] -> 8[0] [receive] via NET/IB/0/GDRDMA
2: jzxh016:896065:896964 [1] NCCL INFO Channel 05/0 : 1[1] -> 9[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896065:896964 [1] NCCL INFO Channel 01/0 : 9[1] -> 1[1] [send] via NET/IB/1/GDRDMA
2: jzxh016:896064:896962 [0] NCCL INFO Channel 04/0 : 4[0] -> 8[0] [receive] via NET/IB/0/GDRDMA
3: jzxh017:51583:52476 [1] NCCL INFO Channel 09/0 : 13[1] -> 5[1] [send] via NET/IB/1/GDRDMA
3: jzxh017:51583:52476 [1] NCCL INFO Channel 13/0 : 13[1] -> 5[1] [send] via NET/IB/1/GDRDMA
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 13/0 : 13[1] -> 5[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 12/0 : 4[0] -> 8[0] [send] via NET/IB/0/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 11/0 : 7[3] -> 11[3] [send] via NET/IB/3/GDRDMA
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 03/0 : 11[3] -> 3[3] [receive] via NET/IB/3/GDRDMA
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 09/0 : 5[1] -> 13[1] [send] via NET/IB/1/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 15/0 : 7[3] -> 11[3] [send] via NET/IB/3/GDRDMA
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 00/0 : 0[0] -> 8[0] [send] via NET/IB/0/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 10/0 : 14[2] -> 6[2] [receive] via NET/IB/2/GDRDMA
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 13/0 : 5[1] -> 13[1] [send] via NET/IB/1/GDRDMA
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 09/0 : 5[1] -> 1[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 14/0 : 14[2] -> 6[2] [receive] via NET/IB/2/GDRDMA
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 07/0 : 11[3] -> 3[3] [receive] via NET/IB/3/GDRDMA
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 04/0 : 0[0] -> 8[0] [send] via NET/IB/0/GDRDMA
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 03/0 : 3[3] -> 11[3] [send] via NET/IB/3/GDRDMA
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 13/0 : 5[1] -> 1[1] [receive] via NET/IB/1/GDRDMA
3: jzxh017:51584:52479 [2] NCCL INFO Channel 10/0 : 6[2] -> 14[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896067:896963 [3] NCCL INFO Channel 07/0 : 7[3] -> 11[3] [receive] via NET/IB/3/GDRDMA
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 07/0 : 3[3] -> 11[3] [send] via NET/IB/3/GDRDMA
3: jzxh017:51584:52479 [2] NCCL INFO Channel 14/0 : 6[2] -> 14[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 02/0 : 2[2] -> 10[2] [receive] via NET/IB/2/GDRDMA
3: jzxh017:51583:52476 [1] NCCL INFO Channel 01/0 : 13[1] -> 9[1] [send] via NET/IB/1/GDRDMA
2: jzxh016:896065:896964 [1] NCCL INFO Channel 05/0 : 9[1] -> 1[1] [send] via NET/IB/1/GDRDMA
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 10/0 : 6[2] -> 2[2] [receive] via NET/IB/2/GDRDMA
3: jzxh017:51584:52479 [2] NCCL INFO Channel 10/0 : 14[2] -> 6[2] [send] via NET/IB/2/GDRDMA
2: jzxh016:896064:896962 [0] NCCL INFO Channel 08/0 : 4[0] -> 8[0] [receive] via NET/IB/0/GDRDMA
2: jzxh016:896067:896963 [3] NCCL INFO Channel 11/0 : 7[3] -> 11[3] [receive] via NET/IB/3/GDRDMA
3: jzxh017:51583:52476 [1] NCCL INFO Channel 05/0 : 13[1] -> 9[1] [send] via NET/IB/1/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 06/0 : 2[2] -> 10[2] [receive] via NET/IB/2/GDRDMA
3: jzxh017:51584:52479 [2] NCCL INFO Channel 14/0 : 14[2] -> 6[2] [send] via NET/IB/2/GDRDMA
2: jzxh016:896067:896963 [3] NCCL INFO Channel 15/0 : 7[3] -> 11[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896064:896962 [0] NCCL INFO Channel 12/0 : 4[0] -> 8[0] [receive] via NET/IB/0/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 02/0 : 10[2] -> 2[2] [send] via NET/IB/2/GDRDMA
2: jzxh016:896067:896963 [3] NCCL INFO Channel 03/0 : 11[3] -> 15[3] [send] via NET/IB/3/GDRDMA
2: jzxh016:896064:896962 [0] NCCL INFO Channel 00/0 : 8[0] -> 12[0] [send] via NET/IB/0/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 06/0 : 10[2] -> 2[2] [send] via NET/IB/2/GDRDMA
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 14/0 : 6[2] -> 2[2] [receive] via NET/IB/2/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 10/0 : 6[2] -> 14[2] [send] via NET/IB/2/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 14/0 : 6[2] -> 14[2] [send] via NET/IB/2/GDRDMA
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 01/0 : 9[1] -> 5[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 05/0 : 9[1] -> 5[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 09/0 : 9[1] -> 5[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896065:896964 [1] NCCL INFO Channel 01/0 : 13[1] -> 9[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896067:896963 [3] NCCL INFO Channel 07/0 : 11[3] -> 15[3] [send] via NET/IB/3/GDRDMA
2: jzxh016:896064:896962 [0] NCCL INFO Channel 04/0 : 8[0] -> 12[0] [send] via NET/IB/0/GDRDMA
2: jzxh016:896065:896964 [1] NCCL INFO Channel 05/0 : 13[1] -> 9[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 02/0 : 14[2] -> 10[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896065:896964 [1] NCCL INFO Channel 01/0 : 9[1] -> 5[1] [send] via NET/IB/1/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 06/0 : 14[2] -> 10[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896065:896964 [1] NCCL INFO Channel 05/0 : 9[1] -> 5[1] [send] via NET/IB/1/GDRDMA
2: jzxh016:896064:896962 [0] NCCL INFO Channel 00/0 : 0[0] -> 8[0] [receive] via NET/IB/0/GDRDMA
2: jzxh016:896067:896963 [3] NCCL INFO Channel 03/0 : 3[3] -> 11[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 02/0 : 10[2] -> 6[2] [send] via NET/IB/2/GDRDMA
3: jzxh017:51584:52479 [2] NCCL INFO Channel 02/0 : 14[2] -> 10[2] [send] via NET/IB/2/GDRDMA
2: jzxh016:896065:896964 [1] NCCL INFO Channel 09/0 : 9[1] -> 5[1] [send] via NET/IB/1/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 08/0 : 12[0] -> 4[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 11/0 : 15[3] -> 7[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896064:896962 [0] NCCL INFO Channel 04/0 : 0[0] -> 8[0] [receive] via NET/IB/0/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 06/0 : 10[2] -> 6[2] [send] via NET/IB/2/GDRDMA
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 13/0 : 9[1] -> 5[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896067:896963 [3] NCCL INFO Channel 07/0 : 3[3] -> 11[3] [receive] via NET/IB/3/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 15/0 : 15[3] -> 7[3] [receive] via NET/IB/3/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 12/0 : 12[0] -> 4[0] [receive] via NET/IB/0/GDRDMA
2: jzxh016:896065:896964 [1] NCCL INFO Channel 13/0 : 9[1] -> 5[1] [send] via NET/IB/1/GDRDMA
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 09/0 : 5[1] -> 1[1] [send] via NET/IB/1/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 02/0 : 10[2] -> 6[2] [receive] via NET/IB/2/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 11/0 : 7[3] -> 15[3] [send] via NET/IB/3/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 08/0 : 4[0] -> 12[0] [send] via NET/IB/0/GDRDMA
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 13/0 : 5[1] -> 1[1] [send] via NET/IB/1/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 15/0 : 7[3] -> 15[3] [send] via NET/IB/3/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 06/0 : 10[2] -> 6[2] [receive] via NET/IB/2/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 12/0 : 4[0] -> 12[0] [send] via NET/IB/0/GDRDMA
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 11/0 : 7[3] -> 3[3] [receive] via NET/IB/3/GDRDMA
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 08/0 : 4[0] -> 0[0] [receive] via NET/IB/0/GDRDMA
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 15/0 : 7[3] -> 3[3] [receive] via NET/IB/3/GDRDMA
0: jzxh014:2356327:2357227 [0] NCCL INFO Channel 12/0 : 4[0] -> 0[0] [receive] via NET/IB/0/GDRDMA
3: jzxh017:51584:52479 [2] NCCL INFO Channel 06/0 : 14[2] -> 10[2] [send] via NET/IB/2/GDRDMA
2: jzxh016:896064:896962 [0] NCCL INFO Channel 00/0 : 8[0] -> 0[0] [send] via NET/IB/0/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 10/0 : 10[2] -> 6[2] [receive] via NET/IB/2/GDRDMA
3: jzxh017:51584:52479 [2] NCCL INFO Channel 02/0 : 14[2] -> 12[0] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 03/0 : 11[3] -> 3[3] [send] via NET/IB/3/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 10/0 : 10[2] -> 6[2] [send] via NET/IB/2/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 14/0 : 10[2] -> 6[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896064:896962 [0] NCCL INFO Channel 04/0 : 8[0] -> 0[0] [send] via NET/IB/0/GDRDMA
2: jzxh016:896067:896963 [3] NCCL INFO Channel 07/0 : 11[3] -> 3[3] [send] via NET/IB/3/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 14/0 : 10[2] -> 6[2] [send] via NET/IB/2/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 10/0 : 6[2] -> 2[2] [send] via NET/IB/2/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 14/0 : 6[2] -> 2[2] [send] via NET/IB/2/GDRDMA
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 02/0 : 2[2] -> 0[0] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 03/0 : 14[2] -> 12[0] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 00/0 : 12[0] -> 8[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 02/0 : 6[2] -> 4[0] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 03/0 : 15[3] -> 11[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896067:896963 [3] NCCL INFO Channel 07/0 : 15[3] -> 11[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896064:896962 [0] NCCL INFO Channel 04/0 : 12[0] -> 8[0] [receive] via NET/IB/0/GDRDMA
2: jzxh016:896067:896963 [3] NCCL INFO Channel 03/0 : 11[3] -> 7[3] [send] via NET/IB/3/GDRDMA
2: jzxh016:896064:896962 [0] NCCL INFO Channel 00/0 : 8[0] -> 4[0] [send] via NET/IB/0/GDRDMA
2: jzxh016:896067:896963 [3] NCCL INFO Channel 07/0 : 11[3] -> 7[3] [send] via NET/IB/3/GDRDMA
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 03/0 : 2[2] -> 0[0] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 04/0 : 8[0] -> 4[0] [send] via NET/IB/0/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 02/0 : 10[2] -> 8[0] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 11/0 : 11[3] -> 7[3] [send] via NET/IB/3/GDRDMA
2: jzxh016:896064:896962 [0] NCCL INFO Channel 08/0 : 8[0] -> 4[0] [send] via NET/IB/0/GDRDMA
2: jzxh016:896067:896963 [3] NCCL INFO Channel 15/0 : 11[3] -> 7[3] [send] via NET/IB/3/GDRDMA
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 05/0 : 2[2] -> 0[0] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 05/0 : 14[2] -> 12[0] via P2P/CUMEM
2: jzxh016:896064:896962 [0] NCCL INFO Channel 12/0 : 8[0] -> 4[0] [send] via NET/IB/0/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 03/0 : 6[2] -> 4[0] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 06/0 : 14[2] -> 12[0] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 03/0 : 10[2] -> 8[0] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 05/0 : 10[2] -> 8[0] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 06/0 : 2[2] -> 0[0] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 10/0 : 14[2] -> 12[0] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 06/0 : 10[2] -> 8[0] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 05/0 : 6[2] -> 4[0] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 10/0 : 2[2] -> 0[0] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 11/0 : 14[2] -> 12[0] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 10/0 : 10[2] -> 8[0] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 06/0 : 6[2] -> 4[0] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 10/0 : 6[2] -> 4[0] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 11/0 : 2[2] -> 0[0] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 11/0 : 7[3] -> 15[3] [receive] via NET/IB/3/GDRDMA
3: jzxh017:51584:52479 [2] NCCL INFO Channel 13/0 : 14[2] -> 12[0] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 15/0 : 7[3] -> 15[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 11/0 : 10[2] -> 8[0] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 11/0 : 6[2] -> 4[0] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 11/0 : 15[3] -> 7[3] [send] via NET/IB/3/GDRDMA
2: jzxh016:896066:896961 [2] NCCL INFO Channel 13/0 : 10[2] -> 8[0] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 13/0 : 6[2] -> 4[0] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 14/0 : 14[2] -> 12[0] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 15/0 : 15[3] -> 7[3] [send] via NET/IB/3/GDRDMA
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 13/0 : 2[2] -> 0[0] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 14/0 : 2[2] -> 0[0] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 14/0 : 10[2] -> 8[0] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 03/0 : 15[3] -> 11[3] [send] via NET/IB/3/GDRDMA
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 14/0 : 6[2] -> 4[0] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 07/0 : 15[3] -> 11[3] [send] via NET/IB/3/GDRDMA
3: jzxh017:51585:52478 [3] NCCL INFO Channel 01/0 : 15[3] -> 12[0] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 03/0 : 11[3] -> 7[3] [receive] via NET/IB/3/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 07/0 : 11[3] -> 7[3] [receive] via NET/IB/3/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 11/0 : 11[3] -> 7[3] [receive] via NET/IB/3/GDRDMA
3: jzxh017:51585:52478 [3] NCCL INFO Channel 04/0 : 15[3] -> 12[0] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 15/0 : 11[3] -> 7[3] [receive] via NET/IB/3/GDRDMA
3: jzxh017:51585:52478 [3] NCCL INFO Channel 05/0 : 15[3] -> 12[0] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 11/0 : 7[3] -> 3[3] [send] via NET/IB/3/GDRDMA
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 15/0 : 7[3] -> 3[3] [send] via NET/IB/3/GDRDMA
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 01/0 : 3[3] -> 0[0] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 01/0 : 11[3] -> 8[0] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 06/0 : 15[3] -> 12[0] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 01/0 : 7[3] -> 4[0] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 09/0 : 15[3] -> 12[0] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 04/0 : 3[3] -> 0[0] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 05/0 : 3[3] -> 0[0] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 04/0 : 11[3] -> 8[0] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 04/0 : 7[3] -> 4[0] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 12/0 : 15[3] -> 12[0] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 06/0 : 3[3] -> 0[0] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 13/0 : 15[3] -> 12[0] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 05/0 : 11[3] -> 8[0] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 05/0 : 7[3] -> 4[0] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 06/0 : 11[3] -> 8[0] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 09/0 : 3[3] -> 0[0] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 09/0 : 11[3] -> 8[0] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 06/0 : 7[3] -> 4[0] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 12/0 : 3[3] -> 0[0] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 13/0 : 3[3] -> 0[0] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 12/0 : 11[3] -> 8[0] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 09/0 : 7[3] -> 4[0] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 14/0 : 3[3] -> 0[0] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 13/0 : 11[3] -> 8[0] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 12/0 : 7[3] -> 4[0] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 14/0 : 15[3] -> 12[0] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 14/0 : 11[3] -> 8[0] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 13/0 : 7[3] -> 4[0] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 14/0 : 7[3] -> 4[0] via P2P/CUMEM
3: jzxh017:51582:52477 [0] NCCL INFO Channel 08/0 : 4[0] -> 12[0] [receive] via NET/IB/0/GDRDMA
3: jzxh017:51582:52477 [0] NCCL INFO Channel 12/0 : 4[0] -> 12[0] [receive] via NET/IB/0/GDRDMA
3: jzxh017:51582:52477 [0] NCCL INFO Channel 08/0 : 12[0] -> 4[0] [send] via NET/IB/0/GDRDMA
3: jzxh017:51582:52477 [0] NCCL INFO Channel 12/0 : 12[0] -> 4[0] [send] via NET/IB/0/GDRDMA
3: jzxh017:51582:52477 [0] NCCL INFO Channel 00/0 : 12[0] -> 8[0] [send] via NET/IB/0/GDRDMA
3: jzxh017:51582:52477 [0] NCCL INFO Channel 04/0 : 12[0] -> 8[0] [send] via NET/IB/0/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 00/0 : 8[0] -> 4[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 04/0 : 8[0] -> 4[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 08/0 : 8[0] -> 4[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 12/0 : 8[0] -> 4[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 08/0 : 4[0] -> 0[0] [send] via NET/IB/0/GDRDMA
1: jzxh015:2965582:2967787 [0] NCCL INFO Channel 12/0 : 4[0] -> 0[0] [send] via NET/IB/0/GDRDMA
3: jzxh017:51585:52478 [3] NCCL INFO Channel 02/0 : 15[3] -> 13[1] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 03/0 : 15[3] -> 13[1] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 05/0 : 15[3] -> 13[1] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 06/0 : 15[3] -> 13[1] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 10/0 : 15[3] -> 13[1] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 02/0 : 3[3] -> 1[1] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 02/0 : 11[3] -> 9[1] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 02/0 : 7[3] -> 5[1] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 11/0 : 15[3] -> 13[1] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 03/0 : 3[3] -> 1[1] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 13/0 : 15[3] -> 13[1] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 03/0 : 7[3] -> 5[1] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 03/0 : 11[3] -> 9[1] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 14/0 : 15[3] -> 13[1] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 05/0 : 3[3] -> 1[1] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 05/0 : 11[3] -> 9[1] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 05/0 : 7[3] -> 5[1] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 00/0 : 15[3] -> 14[2] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 06/0 : 3[3] -> 1[1] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 06/0 : 7[3] -> 5[1] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 04/0 : 15[3] -> 14[2] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 10/0 : 7[3] -> 5[1] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 06/0 : 11[3] -> 9[1] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 11/0 : 7[3] -> 5[1] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 10/0 : 3[3] -> 1[1] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 10/0 : 11[3] -> 9[1] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 07/0 : 15[3] -> 14[2] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 11/0 : 11[3] -> 9[1] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 13/0 : 7[3] -> 5[1] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 11/0 : 3[3] -> 1[1] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 08/0 : 15[3] -> 14[2] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 13/0 : 11[3] -> 9[1] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 14/0 : 7[3] -> 5[1] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 13/0 : 3[3] -> 1[1] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 14/0 : 11[3] -> 9[1] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 14/0 : 3[3] -> 1[1] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 00/0 : 7[3] -> 6[2] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 00/0 : 11[3] -> 10[2] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 12/0 : 15[3] -> 14[2] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 04/0 : 11[3] -> 10[2] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 04/0 : 7[3] -> 6[2] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 07/0 : 11[3] -> 10[2] via P2P/CUMEM
3: jzxh017:51585:52478 [3] NCCL INFO Channel 15/0 : 15[3] -> 14[2] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 07/0 : 7[3] -> 6[2] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 00/0 : 14[2] -> 13[1] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 08/0 : 11[3] -> 10[2] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 01/0 : 14[2] -> 13[1] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 08/0 : 7[3] -> 6[2] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 00/0 : 13[1] -> 12[0] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 12/0 : 11[3] -> 10[2] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 04/0 : 14[2] -> 13[1] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 12/0 : 7[3] -> 6[2] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 03/0 : 13[1] -> 12[0] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 07/0 : 14[2] -> 13[1] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 07/0 : 13[1] -> 12[0] via P2P/CUMEM
0: jzxh014:2356330:2357229 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/CUMEM
2: jzxh016:896067:896963 [3] NCCL INFO Channel 15/0 : 11[3] -> 10[2] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 08/0 : 14[2] -> 13[1] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 08/0 : 13[1] -> 12[0] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 11/0 : 13[1] -> 12[0] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 09/0 : 14[2] -> 13[1] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/CUMEM
1: jzxh015:2965585:2967786 [3] NCCL INFO Channel 15/0 : 7[3] -> 6[2] via P2P/CUMEM
3: jzxh017:51583:52476 [1] NCCL INFO Channel 15/0 : 13[1] -> 12[0] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 12/0 : 14[2] -> 13[1] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 04/0 : 2[2] -> 1[1] via P2P/CUMEM
3: jzxh017:51584:52479 [2] NCCL INFO Channel 15/0 : 14[2] -> 13[1] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 07/0 : 2[2] -> 1[1] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 00/0 : 10[2] -> 9[1] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 08/0 : 2[2] -> 1[1] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 00/0 : 6[2] -> 5[1] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 00/0 : 9[1] -> 8[0] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 01/0 : 10[2] -> 9[1] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 09/0 : 2[2] -> 1[1] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 01/0 : 6[2] -> 5[1] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 03/0 : 9[1] -> 8[0] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 04/0 : 10[2] -> 9[1] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 12/0 : 2[2] -> 1[1] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 04/0 : 6[2] -> 5[1] via P2P/CUMEM
0: jzxh014:2356329:2357228 [2] NCCL INFO Channel 15/0 : 2[2] -> 1[1] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 07/0 : 9[1] -> 8[0] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 07/0 : 10[2] -> 9[1] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 08/0 : 9[1] -> 8[0] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 07/0 : 6[2] -> 5[1] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 08/0 : 10[2] -> 9[1] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 11/0 : 9[1] -> 8[0] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 09/0 : 10[2] -> 9[1] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 00/0 : 5[1] -> 4[0] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 08/0 : 6[2] -> 5[1] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/CUMEM
2: jzxh016:896065:896964 [1] NCCL INFO Channel 15/0 : 9[1] -> 8[0] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 03/0 : 5[1] -> 4[0] via P2P/CUMEM
0: jzxh014:2356328:2357230 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 09/0 : 6[2] -> 5[1] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 12/0 : 10[2] -> 9[1] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO Channel 15/0 : 10[2] -> 9[1] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 07/0 : 5[1] -> 4[0] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 12/0 : 6[2] -> 5[1] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 08/0 : 5[1] -> 4[0] via P2P/CUMEM
1: jzxh015:2965584:2967789 [2] NCCL INFO Channel 15/0 : 6[2] -> 5[1] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 11/0 : 5[1] -> 4[0] via P2P/CUMEM
1: jzxh015:2965583:2967788 [1] NCCL INFO Channel 15/0 : 5[1] -> 4[0] via P2P/CUMEM
0: jzxh014:2356327:2357227 [0] NCCL INFO Connected all trees
0: jzxh014:2356328:2357230 [1] NCCL INFO Connected all trees
0: jzxh014:2356330:2357229 [3] NCCL INFO Connected all trees
0: jzxh014:2356327:2357227 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
0: jzxh014:2356329:2357228 [2] NCCL INFO Connected all trees
0: jzxh014:2356327:2357227 [0] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
0: jzxh014:2356328:2357230 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
0: jzxh014:2356330:2357229 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
0: jzxh014:2356328:2357230 [1] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
0: jzxh014:2356330:2357229 [3] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
0: jzxh014:2356329:2357228 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
0: jzxh014:2356329:2357228 [2] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
1: jzxh015:2965582:2967787 [0] NCCL INFO Connected all trees
1: jzxh015:2965585:2967786 [3] NCCL INFO Connected all trees
1: jzxh015:2965583:2967788 [1] NCCL INFO Connected all trees
1: jzxh015:2965582:2967787 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
1: jzxh015:2965582:2967787 [0] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
1: jzxh015:2965584:2967789 [2] NCCL INFO Connected all trees
1: jzxh015:2965585:2967786 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
2: jzxh016:896064:896962 [0] NCCL INFO Connected all trees
1: jzxh015:2965585:2967786 [3] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
1: jzxh015:2965584:2967789 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
1: jzxh015:2965584:2967789 [2] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
1: jzxh015:2965583:2967788 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
1: jzxh015:2965583:2967788 [1] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
3: jzxh017:51582:52477 [0] NCCL INFO Connected all trees
3: jzxh017:51585:52478 [3] NCCL INFO Connected all trees
3: jzxh017:51582:52477 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
2: jzxh016:896067:896963 [3] NCCL INFO Connected all trees
3: jzxh017:51582:52477 [0] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
3: jzxh017:51585:52478 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
3: jzxh017:51585:52478 [3] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
2: jzxh016:896064:896962 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
2: jzxh016:896064:896962 [0] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
2: jzxh016:896065:896964 [1] NCCL INFO Connected all trees
3: jzxh017:51584:52479 [2] NCCL INFO Connected all trees
3: jzxh017:51583:52476 [1] NCCL INFO Connected all trees
3: jzxh017:51584:52479 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
2: jzxh016:896066:896961 [2] NCCL INFO Connected all trees
2: jzxh016:896067:896963 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
3: jzxh017:51583:52476 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
3: jzxh017:51584:52479 [2] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
3: jzxh017:51583:52476 [1] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
2: jzxh016:896067:896963 [3] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
2: jzxh016:896065:896964 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
2: jzxh016:896066:896961 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
2: jzxh016:896065:896964 [1] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
2: jzxh016:896066:896961 [2] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
1: jzxh015:2965583:2967788 [1] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
1: jzxh015:2965583:2967788 [1] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
1: jzxh015:2965582:2967787 [0] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
1: jzxh015:2965583:2967788 [1] NCCL INFO ncclCommInitRank comm 0x55fffbc332b0 rank 5 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x4261a6b6ec5d7236 - Init COMPLETE
1: jzxh015:2965582:2967787 [0] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
1: jzxh015:2965582:2967787 [0] NCCL INFO ncclCommInitRank comm 0x5581ebd8acb0 rank 4 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x4261a6b6ec5d7236 - Init COMPLETE
1: jzxh015:2965584:2967789 [2] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
1: jzxh015:2965585:2967786 [3] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
1: jzxh015:2965584:2967789 [2] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
1: jzxh015:2965585:2967786 [3] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
1: jzxh015:2965584:2967789 [2] NCCL INFO ncclCommInitRank comm 0x564e0a919f70 rank 6 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x4261a6b6ec5d7236 - Init COMPLETE
1: jzxh015:2965585:2967786 [3] NCCL INFO ncclCommInitRank comm 0x55ee7f078d80 rank 7 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x4261a6b6ec5d7236 - Init COMPLETE
0: jzxh014:2356330:2357229 [3] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
0: jzxh014:2356330:2357229 [3] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
0: jzxh014:2356330:2357229 [3] NCCL INFO ncclCommInitRank comm 0x55abd54283f0 rank 3 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x4261a6b6ec5d7236 - Init COMPLETE
0: jzxh014:2356329:2357228 [2] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
0: jzxh014:2356329:2357228 [2] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
0: jzxh014:2356327:2357227 [0] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
0: jzxh014:2356328:2357230 [1] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
0: jzxh014:2356328:2357230 [1] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
0: jzxh014:2356327:2357227 [0] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
0: jzxh014:2356329:2357228 [2] NCCL INFO ncclCommInitRank comm 0x55e468be3960 rank 2 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x4261a6b6ec5d7236 - Init COMPLETE
0: jzxh014:2356328:2357230 [1] NCCL INFO ncclCommInitRank comm 0x559d12187a60 rank 1 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x4261a6b6ec5d7236 - Init COMPLETE
0: jzxh014:2356327:2357227 [0] NCCL INFO ncclCommInitRank comm 0x555ba8ca5800 rank 0 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x4261a6b6ec5d7236 - Init COMPLETE
3: jzxh017:51584:52479 [2] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
3: jzxh017:51584:52479 [2] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
3: jzxh017:51584:52479 [2] NCCL INFO ncclCommInitRank comm 0x564cbfab3310 rank 14 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x4261a6b6ec5d7236 - Init COMPLETE
3: jzxh017:51585:52478 [3] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
3: jzxh017:51585:52478 [3] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
3: jzxh017:51585:52478 [3] NCCL INFO ncclCommInitRank comm 0x56086fae1720 rank 15 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x4261a6b6ec5d7236 - Init COMPLETE
3: jzxh017:51583:52476 [1] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
3: jzxh017:51582:52477 [0] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
3: jzxh017:51583:52476 [1] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
3: jzxh017:51582:52477 [0] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
3: jzxh017:51583:52476 [1] NCCL INFO ncclCommInitRank comm 0x558abda268d0 rank 13 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x4261a6b6ec5d7236 - Init COMPLETE
3: jzxh017:51582:52477 [0] NCCL INFO ncclCommInitRank comm 0x559e3dba4770 rank 12 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x4261a6b6ec5d7236 - Init COMPLETE
1: jzxh015:2965584:2967818 [2] NCCL INFO Channel 12/1 : 6[2] -> 0[0] [send] via NET/IB/0(4)/GDRDMA/Shared
1: jzxh015:2965584:2967818 [2] NCCL INFO Channel 13/1 : 6[2] -> 0[0] [send] via NET/IB/0(4)/GDRDMA/Shared
1: jzxh015:2965582:2967819 [0] NCCL INFO Channel 12/1 : 4[0] -> 0[0] [send] via NET/IB/0/GDRDMA/Shared
1: jzxh015:2965582:2967819 [0] NCCL INFO Channel 13/1 : 4[0] -> 0[0] [send] via NET/IB/0/GDRDMA/Shared
1: jzxh015:2965585:2967820 [3] NCCL INFO Channel 12/1 : 7[3] -> 0[0] [send] via NET/IB/0(4)/GDRDMA/Shared
1: jzxh015:2965583:2967821 [1] NCCL INFO Channel 12/1 : 5[1] -> 0[0] [send] via NET/IB/0(4)/GDRDMA/Shared
1: jzxh015:2965583:2967821 [1] NCCL INFO Channel 13/1 : 5[1] -> 0[0] [send] via NET/IB/0(4)/GDRDMA/Shared
0: jzxh014:2356329:2357261 [2] NCCL INFO Channel 00/1 : 2[2] -> 0[0] via P2P/CUMEM
1: jzxh015:2965585:2967820 [3] NCCL INFO Channel 13/1 : 7[3] -> 0[0] [send] via NET/IB/0(4)/GDRDMA/Shared
0: jzxh014:2356327:2357262 [0] NCCL INFO Channel 08/1 : 15[3] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
0: jzxh014:2356330:2357260 [3] NCCL INFO Channel 00/1 : 3[3] -> 0[0] via P2P/CUMEM
0: jzxh014:2356327:2357262 [0] NCCL INFO Channel 09/1 : 15[3] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
0: jzxh014:2356328:2357259 [1] NCCL INFO Channel 00/1 : 1[1] -> 0[0] via P2P/CUMEM
0: jzxh014:2356329:2357261 [2] NCCL INFO Channel 01/1 : 2[2] -> 0[0] via P2P/CUMEM
0: jzxh014:2356330:2357260 [3] NCCL INFO Channel 01/1 : 3[3] -> 0[0] via P2P/CUMEM
3: jzxh017:51583:52508 [1] NCCL INFO Channel 08/1 : 13[1] -> 0[0] [send] via NET/IB/0(12)/GDRDMA/Shared
2: jzxh016:896065:896964 [1] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
2: jzxh016:896065:896964 [1] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
2: jzxh016:896065:896964 [1] NCCL INFO ncclCommInitRank comm 0x55a46b9ae900 rank 9 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x4261a6b6ec5d7236 - Init COMPLETE
3: jzxh017:51583:52508 [1] NCCL INFO Channel 09/1 : 13[1] -> 0[0] [send] via NET/IB/0(12)/GDRDMA/Shared
0: jzxh014:2356328:2357259 [1] NCCL INFO Channel 01/1 : 1[1] -> 0[0] via P2P/CUMEM
2: jzxh016:896066:896961 [2] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
2: jzxh016:896067:896963 [3] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
2: jzxh016:896066:896961 [2] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
2: jzxh016:896064:896962 [0] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
2: jzxh016:896067:896963 [3] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
2: jzxh016:896066:896961 [2] NCCL INFO ncclCommInitRank comm 0x556ee6361e10 rank 10 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x4261a6b6ec5d7236 - Init COMPLETE
2: jzxh016:896064:896962 [0] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
2: jzxh016:896067:896963 [3] NCCL INFO ncclCommInitRank comm 0x5576b8f9bbe0 rank 11 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x4261a6b6ec5d7236 - Init COMPLETE
2: jzxh016:896064:896962 [0] NCCL INFO ncclCommInitRank comm 0x55a50cad5640 rank 8 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x4261a6b6ec5d7236 - Init COMPLETE
3: jzxh017:51585:52510 [3] NCCL INFO Channel 08/1 : 15[3] -> 0[0] [send] via NET/IB/0(12)/GDRDMA/Shared
3: jzxh017:51582:52509 [0] NCCL INFO Channel 08/1 : 12[0] -> 0[0] [send] via NET/IB/0/GDRDMA/Shared
0: jzxh014:2356327:2357262 [0] NCCL INFO Channel 08/1 : 14[2] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
0: jzxh014:2356327:2357262 [0] NCCL INFO Channel 09/1 : 14[2] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
3: jzxh017:51585:52510 [3] NCCL INFO Channel 09/1 : 15[3] -> 0[0] [send] via NET/IB/0(12)/GDRDMA/Shared
3: jzxh017:51582:52509 [0] NCCL INFO Channel 09/1 : 12[0] -> 0[0] [send] via NET/IB/0/GDRDMA/Shared
3: jzxh017:51584:52511 [2] NCCL INFO Channel 08/1 : 14[2] -> 0[0] [send] via NET/IB/0(12)/GDRDMA/Shared
3: jzxh017:51584:52511 [2] NCCL INFO Channel 09/1 : 14[2] -> 0[0] [send] via NET/IB/0(12)/GDRDMA/Shared
0: jzxh014:2356327:2357262 [0] NCCL INFO Channel 08/1 : 13[1] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
0: jzxh014:2356327:2357262 [0] NCCL INFO Channel 09/1 : 13[1] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
0: jzxh014:2356327:2357262 [0] NCCL INFO Channel 08/1 : 12[0] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
0: jzxh014:2356327:2357262 [0] NCCL INFO Channel 09/1 : 12[0] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
0: jzxh014:2356327:2357262 [0] NCCL INFO Channel 04/1 : 11[3] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
0: jzxh014:2356327:2357262 [0] NCCL INFO Channel 05/1 : 11[3] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
2: jzxh016:896065:896993 [1] NCCL INFO Channel 04/1 : 9[1] -> 0[0] [send] via NET/IB/0(8)/GDRDMA/Shared
2: jzxh016:896067:896994 [3] NCCL INFO Channel 04/1 : 11[3] -> 0[0] [send] via NET/IB/0(8)/GDRDMA/Shared
2: jzxh016:896065:896993 [1] NCCL INFO Channel 05/1 : 9[1] -> 0[0] [send] via NET/IB/0(8)/GDRDMA/Shared
2: jzxh016:896067:896994 [3] NCCL INFO Channel 05/1 : 11[3] -> 0[0] [send] via NET/IB/0(8)/GDRDMA/Shared
0: jzxh014:2356327:2357262 [0] NCCL INFO Channel 04/1 : 10[2] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
0: jzxh014:2356327:2357262 [0] NCCL INFO Channel 05/1 : 10[2] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
2: jzxh016:896066:896995 [2] NCCL INFO Channel 04/1 : 10[2] -> 0[0] [send] via NET/IB/0(8)/GDRDMA/Shared
2: jzxh016:896066:896995 [2] NCCL INFO Channel 05/1 : 10[2] -> 0[0] [send] via NET/IB/0(8)/GDRDMA/Shared
2: jzxh016:896064:896996 [0] NCCL INFO Channel 04/1 : 8[0] -> 0[0] [send] via NET/IB/0/GDRDMA/Shared
0: jzxh014:2356327:2357262 [0] NCCL INFO Channel 04/1 : 9[1] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
0: jzxh014:2356327:2357262 [0] NCCL INFO Channel 05/1 : 9[1] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
0: jzxh014:2356327:2357262 [0] NCCL INFO Channel 04/1 : 8[0] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
0: jzxh014:2356327:2357262 [0] NCCL INFO Channel 05/1 : 8[0] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
2: jzxh016:896064:896996 [0] NCCL INFO Channel 05/1 : 8[0] -> 0[0] [send] via NET/IB/0/GDRDMA/Shared
0: jzxh014:2356327:2357262 [0] NCCL INFO Channel 12/1 : 7[3] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
0: jzxh014:2356327:2357262 [0] NCCL INFO Channel 13/1 : 7[3] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
0: jzxh014:2356327:2357262 [0] NCCL INFO Channel 12/1 : 6[2] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
0: jzxh014:2356327:2357262 [0] NCCL INFO Channel 13/1 : 6[2] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
0: jzxh014:2356327:2357262 [0] NCCL INFO Channel 12/1 : 5[1] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
0: jzxh014:2356327:2357262 [0] NCCL INFO Channel 13/1 : 5[1] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
0: jzxh014:2356327:2357262 [0] NCCL INFO Channel 12/1 : 4[0] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
0: jzxh014:2356327:2357262 [0] NCCL INFO Channel 13/1 : 4[0] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
0: [2025-11-23 14:47:06,273] [INFO] [axolotl.utils.samplers.multipack.calc_min_len:436] [PID:2356327] [RANK:0] gather_len_batches: [79586, 79586, 79586, 79586, 79586, 79586, 79586, 79586, 79586, 79586, 79586, 79586, 79586, 79586, 79586, 79586]
0: [2025-11-23 14:47:06,314] [INFO] [axolotl.utils.trainer.calc_sample_packing_eff_est:495] [PID:2356327] [RANK:0] sample_packing_eff_est across ranks: [0.8591607213020325, 0.8591607213020325, 0.8591607213020325, 0.8591607213020325, 0.8591607213020325, 0.8591607213020325, 0.8591607213020325, 0.8591607213020325, 0.8591607213020325, 0.8591607213020325, 0.8591607213020325, 0.8591607213020325, 0.8591607213020325, 0.8591607213020325, 0.8591607213020325, 0.8591607213020325]
0: [2025-11-23 14:47:06,331] [INFO] [axolotl.utils.data.sft._prepare_standard_dataset:127] [PID:2356327] [RANK:0] Maximum number of steps set at 2984
1: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
3: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
3: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
3: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
0: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
2: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
1: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
0: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
2: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
0: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
0: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
1: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
3: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
2: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
2: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
1: Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
0: [2025-11-23 14:47:13,632] [INFO] [axolotl.monkeypatch.transformers.trainer_loss_calc.patch_evaluation_loop:110] [PID:2356327] [RANK:0] Patched Trainer.evaluation_loop with nanmean loss calculation
0: [2025-11-23 14:47:13,634] [INFO] [axolotl.monkeypatch.transformers.trainer_loss_calc.patch_maybe_log_save_evaluate:164] [PID:2356327] [RANK:0] Patched Trainer._maybe_log_save_evaluate with nanmean loss calculation
2: 
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:10<00:10, 10.01s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:10<00:10, 10.01s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:10<00:10, 10.01s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:10<00:10, 10.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.14s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.57s/it]
1: 
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:10<00:10, 10.01s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:10<00:10, 10.01s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:10<00:10, 10.01s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:10<00:10, 10.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.14s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.57s/it]
1: 
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.14s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.57s/it]
3: 
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:10<00:10, 10.01s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:10<00:10, 10.01s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:10<00:10, 10.01s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:10<00:10, 10.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.14s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.57s/it]
2: 
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.14s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.57s/it]
2: 
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.14s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.57s/it]
1: 
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.14s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.57s/it]
3: 
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.14s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.57s/it]
1: 
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.14s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.57s/it]
3: 
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.14s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.57s/it]
2: 
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.14s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.57s/it]
0: 
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:10<00:10, 10.01s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:10<00:10, 10.01s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:10<00:10, 10.01s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:10<00:10, 10.11s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.14s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.57s/it]
3: 
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.14s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.57s/it]
0: 
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.14s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.57s/it]
0: 
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.14s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.57s/it]
0: 
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.15s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:15<00:00,  7.60s/it]
0: [2025-11-23 14:47:33,984] [INFO] [axolotl.loaders.model._configure_embedding_dtypes:345] [PID:2356327] [RANK:0] Converting modules to torch.bfloat16
0: [2025-11-23 14:47:37,860] [INFO] [axolotl.train.save_initial_configs:416] [PID:2356327] [RANK:0] Pre-saving tokenizer to /lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-4b/0.75...
0: [2025-11-23 14:47:38,542] [INFO] [axolotl.train.save_initial_configs:419] [PID:2356327] [RANK:0] Pre-saving model config to /lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-4b/0.75...
0: [2025-11-23 14:47:38,556] [INFO] [axolotl.train.save_initial_configs:423] [PID:2356327] [RANK:0] Pre-saving processor to /lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-4b/0.75...
0: [2025-11-23 14:47:41,670] [INFO] [axolotl.train.execute_training:203] [PID:2356327] [RANK:0] Starting trainer...
0: [2025-11-23 14:52:43,878] [INFO] [axolotl.utils.samplers.multipack.calc_min_len:436] [PID:2356327] [RANK:0] gather_len_batches: [79586, 79586, 79586, 79586, 79586, 79586, 79586, 79586, 79586, 79586, 79586, 79586, 79586, 79586, 79586, 79586]
0: Parameter Offload - Persistent parameters statistics: param_count = 479, numel = 768880
0: jzxh014:2356327:2358108 [0] NCCL INFO Comm config Blocking set to 1
0: jzxh014:2356329:2358117 [2] NCCL INFO Comm config Blocking set to 1
0: jzxh014:2356328:2358113 [1] NCCL INFO Comm config Blocking set to 1
1: jzxh015:2965585:2968660 [3] NCCL INFO Comm config Blocking set to 1
2: jzxh016:896065:897829 [1] NCCL INFO Comm config Blocking set to 1
2: jzxh016:896066:897833 [2] NCCL INFO Comm config Blocking set to 1
2: jzxh016:896064:897825 [0] NCCL INFO Comm config Blocking set to 1
0: jzxh014:2356329:2358125 [2] NCCL INFO Using non-device net plugin version 0
0: jzxh014:2356329:2358125 [2] NCCL INFO Using network IB
0: jzxh014:2356328:2358126 [1] NCCL INFO Using non-device net plugin version 0
0: jzxh014:2356328:2358126 [1] NCCL INFO Using network IB
0: jzxh014:2356327:2358124 [0] NCCL INFO Using non-device net plugin version 0
0: jzxh014:2356327:2358124 [0] NCCL INFO Using network IB
0: jzxh014:2356330:2358121 [3] NCCL INFO Comm config Blocking set to 1
3: jzxh017:51585:53366 [3] NCCL INFO Comm config Blocking set to 1
1: jzxh015:2965584:2968656 [2] NCCL INFO Comm config Blocking set to 1
3: jzxh017:51584:53357 [2] NCCL INFO Comm config Blocking set to 1
3: jzxh017:51583:53356 [1] NCCL INFO Comm config Blocking set to 1
2: jzxh016:896064:897841 [0] NCCL INFO Using non-device net plugin version 0
2: jzxh016:896065:897839 [1] NCCL INFO Using non-device net plugin version 0
2: jzxh016:896067:897838 [3] NCCL INFO Comm config Blocking set to 1
2: jzxh016:896064:897841 [0] NCCL INFO Using network IB
2: jzxh016:896065:897839 [1] NCCL INFO Using network IB
0: jzxh014:2356330:2358127 [3] NCCL INFO Using non-device net plugin version 0
0: jzxh014:2356330:2358127 [3] NCCL INFO Using network IB
2: jzxh016:896066:897840 [2] NCCL INFO Using non-device net plugin version 0
2: jzxh016:896066:897840 [2] NCCL INFO Using network IB
1: jzxh015:2965583:2968655 [1] NCCL INFO Comm config Blocking set to 1
1: jzxh015:2965585:2968664 [3] NCCL INFO Using non-device net plugin version 0
1: jzxh015:2965585:2968664 [3] NCCL INFO Using network IB
1: jzxh015:2965584:2968665 [2] NCCL INFO Using non-device net plugin version 0
1: jzxh015:2965584:2968665 [2] NCCL INFO Using network IB
3: jzxh017:51585:53367 [3] NCCL INFO Using non-device net plugin version 0
3: jzxh017:51585:53367 [3] NCCL INFO Using network IB
3: jzxh017:51584:53368 [2] NCCL INFO Using non-device net plugin version 0
3: jzxh017:51584:53368 [2] NCCL INFO Using network IB
3: jzxh017:51583:53369 [1] NCCL INFO Using non-device net plugin version 0
3: jzxh017:51583:53369 [1] NCCL INFO Using network IB
3: jzxh017:51582:53353 [0] NCCL INFO Comm config Blocking set to 1
1: jzxh015:2965582:2968648 [0] NCCL INFO Comm config Blocking set to 1
1: jzxh015:2965583:2968666 [1] NCCL INFO Using non-device net plugin version 0
1: jzxh015:2965583:2968666 [1] NCCL INFO Using network IB
3: jzxh017:51582:53370 [0] NCCL INFO Using non-device net plugin version 0
3: jzxh017:51582:53370 [0] NCCL INFO Using network IB
2: jzxh016:896067:897842 [3] NCCL INFO Using non-device net plugin version 0
2: jzxh016:896067:897842 [3] NCCL INFO Using network IB
1: jzxh015:2965582:2968667 [0] NCCL INFO Using non-device net plugin version 0
1: jzxh015:2965582:2968667 [0] NCCL INFO Using network IB
0: jzxh014:2356330:2358127 [3] NCCL INFO ncclCommInitRank comm 0x14c18412ee00 rank 3 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x619ce177681b4aa5 - Init START
0: jzxh014:2356329:2358125 [2] NCCL INFO ncclCommInitRank comm 0x153460122200 rank 2 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x619ce177681b4aa5 - Init START
0: jzxh014:2356328:2358126 [1] NCCL INFO ncclCommInitRank comm 0x15475412d200 rank 1 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x619ce177681b4aa5 - Init START
0: jzxh014:2356327:2358124 [0] NCCL INFO ncclCommInitRank comm 0x1472b8130d20 rank 0 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x619ce177681b4aa5 - Init START
3: jzxh017:51582:53370 [0] NCCL INFO ncclCommInitRank comm 0x14552c11e540 rank 12 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x619ce177681b4aa5 - Init START
3: jzxh017:51583:53369 [1] NCCL INFO ncclCommInitRank comm 0x14b65c1450c0 rank 13 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x619ce177681b4aa5 - Init START
3: jzxh017:51584:53368 [2] NCCL INFO ncclCommInitRank comm 0x14ca90136dc0 rank 14 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x619ce177681b4aa5 - Init START
3: jzxh017:51585:53367 [3] NCCL INFO ncclCommInitRank comm 0x1459cc121240 rank 15 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x619ce177681b4aa5 - Init START
2: jzxh016:896067:897842 [3] NCCL INFO ncclCommInitRank comm 0x14a628114540 rank 11 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x619ce177681b4aa5 - Init START
2: jzxh016:896064:897841 [0] NCCL INFO ncclCommInitRank comm 0x15476812cf80 rank 8 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x619ce177681b4aa5 - Init START
2: jzxh016:896065:897839 [1] NCCL INFO ncclCommInitRank comm 0x14f2fc1412c0 rank 9 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x619ce177681b4aa5 - Init START
2: jzxh016:896066:897840 [2] NCCL INFO ncclCommInitRank comm 0x1498d4139700 rank 10 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x619ce177681b4aa5 - Init START
1: jzxh015:2965585:2968664 [3] NCCL INFO ncclCommInitRank comm 0x145b10135ed0 rank 7 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x619ce177681b4aa5 - Init START
1: jzxh015:2965584:2968665 [2] NCCL INFO ncclCommInitRank comm 0x149484114480 rank 6 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x619ce177681b4aa5 - Init START
1: jzxh015:2965582:2968667 [0] NCCL INFO ncclCommInitRank comm 0x151e3812df00 rank 4 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x619ce177681b4aa5 - Init START
1: jzxh015:2965583:2968666 [1] NCCL INFO ncclCommInitRank comm 0x14e36c136200 rank 5 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x619ce177681b4aa5 - Init START
3: jzxh017:51585:53367 [3] NCCL INFO Setting affinity for GPU 3 to ffffff00,00000000,00000000,ffffff00,00000000,00000000
3: jzxh017:51585:53367 [3] NCCL INFO NVLS multicast support is not available on dev 3
2: jzxh016:896065:897839 [1] NCCL INFO Setting affinity for GPU 1 to ffff,ff000000,00000000,0000ffff,ff000000
2: jzxh016:896065:897839 [1] NCCL INFO NVLS multicast support is not available on dev 1
3: jzxh017:51584:53368 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffff0000,00000000,000000ff,ffff0000,00000000
3: jzxh017:51584:53368 [2] NCCL INFO NVLS multicast support is not available on dev 2
3: jzxh017:51583:53369 [1] NCCL INFO Setting affinity for GPU 1 to ffff,ff000000,00000000,0000ffff,ff000000
3: jzxh017:51583:53369 [1] NCCL INFO NVLS multicast support is not available on dev 1
3: jzxh017:51582:53370 [0] NCCL INFO Setting affinity for GPU 0 to ffffff,00000000,00000000,00ffffff
3: jzxh017:51582:53370 [0] NCCL INFO NVLS multicast support is not available on dev 0
0: jzxh014:2356328:2358126 [1] NCCL INFO Setting affinity for GPU 1 to ffff,ff000000,00000000,0000ffff,ff000000
0: jzxh014:2356328:2358126 [1] NCCL INFO NVLS multicast support is not available on dev 1
1: jzxh015:2965585:2968664 [3] NCCL INFO Setting affinity for GPU 3 to ffffff00,00000000,00000000,ffffff00,00000000,00000000
1: jzxh015:2965585:2968664 [3] NCCL INFO NVLS multicast support is not available on dev 3
0: jzxh014:2356330:2358127 [3] NCCL INFO Setting affinity for GPU 3 to ffffff00,00000000,00000000,ffffff00,00000000,00000000
0: jzxh014:2356330:2358127 [3] NCCL INFO NVLS multicast support is not available on dev 3
2: jzxh016:896066:897840 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffff0000,00000000,000000ff,ffff0000,00000000
2: jzxh016:896066:897840 [2] NCCL INFO NVLS multicast support is not available on dev 2
2: jzxh016:896064:897841 [0] NCCL INFO Setting affinity for GPU 0 to ffffff,00000000,00000000,00ffffff
0: jzxh014:2356329:2358125 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffff0000,00000000,000000ff,ffff0000,00000000
0: jzxh014:2356329:2358125 [2] NCCL INFO NVLS multicast support is not available on dev 2
0: jzxh014:2356327:2358124 [0] NCCL INFO Setting affinity for GPU 0 to ffffff,00000000,00000000,00ffffff
2: jzxh016:896064:897841 [0] NCCL INFO NVLS multicast support is not available on dev 0
0: jzxh014:2356327:2358124 [0] NCCL INFO NVLS multicast support is not available on dev 0
2: jzxh016:896067:897842 [3] NCCL INFO Setting affinity for GPU 3 to ffffff00,00000000,00000000,ffffff00,00000000,00000000
2: jzxh016:896067:897842 [3] NCCL INFO NVLS multicast support is not available on dev 3
1: jzxh015:2965583:2968666 [1] NCCL INFO Setting affinity for GPU 1 to ffff,ff000000,00000000,0000ffff,ff000000
1: jzxh015:2965583:2968666 [1] NCCL INFO NVLS multicast support is not available on dev 1
1: jzxh015:2965582:2968667 [0] NCCL INFO Setting affinity for GPU 0 to ffffff,00000000,00000000,00ffffff
1: jzxh015:2965582:2968667 [0] NCCL INFO NVLS multicast support is not available on dev 0
1: jzxh015:2965584:2968665 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffff0000,00000000,000000ff,ffff0000,00000000
1: jzxh015:2965584:2968665 [2] NCCL INFO NVLS multicast support is not available on dev 2
3: jzxh017:51582:53370 [0] NCCL INFO comm 0x14552c11e540 rank 12 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0
3: jzxh017:51585:53367 [3] NCCL INFO comm 0x1459cc121240 rank 15 nRanks 16 nNodes 4 localRanks 4 localRank 3 MNNVL 0
3: jzxh017:51583:53369 [1] NCCL INFO comm 0x14b65c1450c0 rank 13 nRanks 16 nNodes 4 localRanks 4 localRank 1 MNNVL 0
3: jzxh017:51584:53368 [2] NCCL INFO comm 0x14ca90136dc0 rank 14 nRanks 16 nNodes 4 localRanks 4 localRank 2 MNNVL 0
3: jzxh017:51582:53370 [0] NCCL INFO Trees [0] 13/-1/-1->12->8 [1] -1/-1/-1->12->15 [2] 13/-1/-1->12->14 [3] 14/-1/-1->12->13 [4] 15/-1/-1->12->8 [5] 14/-1/-1->12->15 [6] 15/-1/-1->12->14 [7] -1/-1/-1->12->13 [8] 13/4/-1->12->-1 [9] -1/-1/-1->12->15 [10] 13/-1/-1->12->14 [11] 14/-1/-1->12->13 [12] 15/4/-1->12->-1 [13] 14/-1/-1->12->15 [14] 15/-1/-1->12->14 [15] -1/-1/-1->12->13
3: jzxh017:51582:53370 [0] NCCL INFO P2P Chunksize set to 131072
0: jzxh014:2356330:2358127 [3] NCCL INFO comm 0x14c18412ee00 rank 3 nRanks 16 nNodes 4 localRanks 4 localRank 3 MNNVL 0
0: jzxh014:2356329:2358125 [2] NCCL INFO comm 0x153460122200 rank 2 nRanks 16 nNodes 4 localRanks 4 localRank 2 MNNVL 0
0: jzxh014:2356328:2358126 [1] NCCL INFO comm 0x15475412d200 rank 1 nRanks 16 nNodes 4 localRanks 4 localRank 1 MNNVL 0
0: jzxh014:2356330:2358127 [3] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] 0/-1/-1->3->2 [2] -1/-1/-1->3->1 [3] 1/11/-1->3->-1 [4] 2/-1/-1->3->0 [5] 0/-1/-1->3->1 [6] 1/-1/-1->3->0 [7] 2/11/-1->3->-1 [8] -1/-1/-1->3->2 [9] 0/-1/-1->3->2 [10] -1/-1/-1->3->1 [11] 1/-1/-1->3->7 [12] 2/-1/-1->3->0 [13] 0/-1/-1->3->1 [14] 1/-1/-1->3->0 [15] 2/-1/-1->3->7
0: jzxh014:2356329:2358125 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 0/10/-1->2->-1 [3] -1/-1/-1->2->0 [4] 1/-1/-1->2->3 [5] -1/-1/-1->2->0 [6] 0/10/-1->2->-1 [7] 1/-1/-1->2->3 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 0/-1/-1->2->6 [11] -1/-1/-1->2->0 [12] 1/-1/-1->2->3 [13] -1/-1/-1->2->0 [14] 0/-1/-1->2->6 [15] 1/-1/-1->2->3
0: jzxh014:2356330:2358127 [3] NCCL INFO P2P Chunksize set to 131072
0: jzxh014:2356329:2358125 [2] NCCL INFO P2P Chunksize set to 131072
0: jzxh014:2356327:2358124 [0] NCCL INFO comm 0x1472b8130d20 rank 0 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0
0: jzxh014:2356328:2358126 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/9/-1->1->-1 [2] 3/-1/-1->1->0 [3] 0/-1/-1->1->3 [4] -1/-1/-1->1->2 [5] 3/9/-1->1->-1 [6] -1/-1/-1->1->3 [7] 0/-1/-1->1->2 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->5 [10] 3/-1/-1->1->0 [11] 0/-1/-1->1->3 [12] -1/-1/-1->1->2 [13] 3/-1/-1->1->5 [14] -1/-1/-1->1->3 [15] 0/-1/-1->1->2
0: jzxh014:2356328:2358126 [1] NCCL INFO P2P Chunksize set to 131072
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 00/16 :    0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 01/16 :    0   3   2   5   4   7   6   9   8  11  10  13  12  15  14   1
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 02/16 :    0   3   6   5   4   7  10   9   8  11  14  13  12  15   2   1
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 03/16 :    0   1   2   7   4   5   6  11   8   9  10  15  12  13  14   3
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 04/16 :    0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 05/16 :    0   3   2   5   4   7   6   9   8  11  10  13  12  15  14   1
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 06/16 :    0   3   6   5   4   7  10   9   8  11  14  13  12  15   2   1
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 07/16 :    0   1   2   7   4   5   6  11   8   9  10  15  12  13  14   3
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 08/16 :    0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 09/16 :    0   3   2   5   4   7   6   9   8  11  10  13  12  15  14   1
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 10/16 :    0   3   6   5   4   7  10   9   8  11  14  13  12  15   2   1
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 11/16 :    0   1   2   7   4   5   6  11   8   9  10  15  12  13  14   3
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 12/16 :    0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 13/16 :    0   3   2   5   4   7   6   9   8  11  10  13  12  15  14   1
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 14/16 :    0   3   6   5   4   7  10   9   8  11  14  13  12  15   2   1
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 15/16 :    0   1   2   7   4   5   6  11   8   9  10  15  12  13  14   3
3: jzxh017:51585:53367 [3] NCCL INFO Trees [0] -1/-1/-1->15->14 [1] 12/-1/-1->15->14 [2] -1/-1/-1->15->13 [3] 13/-1/-1->15->11 [4] 14/-1/-1->15->12 [5] 12/-1/-1->15->13 [6] 13/-1/-1->15->12 [7] 14/-1/-1->15->11 [8] -1/-1/-1->15->14 [9] 12/-1/-1->15->14 [10] -1/-1/-1->15->13 [11] 13/7/-1->15->-1 [12] 14/-1/-1->15->12 [13] 12/-1/-1->15->13 [14] 13/-1/-1->15->12 [15] 14/7/-1->15->-1
3: jzxh017:51583:53369 [1] NCCL INFO Trees [0] 14/-1/-1->13->12 [1] 14/-1/-1->13->9 [2] 15/-1/-1->13->12 [3] 12/-1/-1->13->15 [4] -1/-1/-1->13->14 [5] 15/-1/-1->13->9 [6] -1/-1/-1->13->15 [7] 12/-1/-1->13->14 [8] 14/-1/-1->13->12 [9] 14/5/-1->13->-1 [10] 15/-1/-1->13->12 [11] 12/-1/-1->13->15 [12] -1/-1/-1->13->14 [13] 15/5/-1->13->-1 [14] -1/-1/-1->13->15 [15] 12/-1/-1->13->14
3: jzxh017:51585:53367 [3] NCCL INFO P2P Chunksize set to 131072
2: jzxh016:896067:897842 [3] NCCL INFO comm 0x14a628114540 rank 11 nRanks 16 nNodes 4 localRanks 4 localRank 3 MNNVL 0
2: jzxh016:896066:897840 [2] NCCL INFO comm 0x1498d4139700 rank 10 nRanks 16 nNodes 4 localRanks 4 localRank 2 MNNVL 0
2: jzxh016:896067:897842 [3] NCCL INFO Trees [0] -1/-1/-1->11->10 [1] 8/-1/-1->11->10 [2] -1/-1/-1->11->9 [3] 9/7/15->11->3 [4] 10/-1/-1->11->8 [5] 8/-1/-1->11->9 [6] 9/-1/-1->11->8 [7] 10/7/15->11->3 [8] -1/-1/-1->11->10 [9] 8/-1/-1->11->10 [10] -1/-1/-1->11->9 [11] 9/-1/-1->11->7 [12] 10/-1/-1->11->8 [13] 8/-1/-1->11->9 [14] 9/-1/-1->11->8 [15] 10/-1/-1->11->7
2: jzxh016:896066:897840 [2] NCCL INFO Trees [0] 11/-1/-1->10->9 [1] 11/-1/-1->10->9 [2] 8/6/14->10->2 [3] -1/-1/-1->10->8 [4] 9/-1/-1->10->11 [5] -1/-1/-1->10->8 [6] 8/6/14->10->2 [7] 9/-1/-1->10->11 [8] 11/-1/-1->10->9 [9] 11/-1/-1->10->9 [10] 8/-1/-1->10->6 [11] -1/-1/-1->10->8 [12] 9/-1/-1->10->11 [13] -1/-1/-1->10->8 [14] 8/-1/-1->10->6 [15] 9/-1/-1->10->11
1: jzxh015:2965585:2968664 [3] NCCL INFO comm 0x145b10135ed0 rank 7 nRanks 16 nNodes 4 localRanks 4 localRank 3 MNNVL 0
1: jzxh015:2965582:2968667 [0] NCCL INFO comm 0x151e3812df00 rank 4 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0
1: jzxh015:2965583:2968666 [1] NCCL INFO comm 0x14e36c136200 rank 5 nRanks 16 nNodes 4 localRanks 4 localRank 1 MNNVL 0
1: jzxh015:2965584:2968665 [2] NCCL INFO comm 0x149484114480 rank 6 nRanks 16 nNodes 4 localRanks 4 localRank 2 MNNVL 0
1: jzxh015:2965585:2968664 [3] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] 4/-1/-1->7->6 [2] -1/-1/-1->7->5 [3] 5/-1/-1->7->11 [4] 6/-1/-1->7->4 [5] 4/-1/-1->7->5 [6] 5/-1/-1->7->4 [7] 6/-1/-1->7->11 [8] -1/-1/-1->7->6 [9] 4/-1/-1->7->6 [10] -1/-1/-1->7->5 [11] 5/11/3->7->15 [12] 6/-1/-1->7->4 [13] 4/-1/-1->7->5 [14] 5/-1/-1->7->4 [15] 6/11/3->7->15
1: jzxh015:2965585:2968664 [3] NCCL INFO P2P Chunksize set to 131072
0: jzxh014:2356327:2358124 [0] NCCL INFO Trees [0] 1/8/-1->0->-1 [1] -1/-1/-1->0->3 [2] 1/-1/-1->0->2 [3] 2/-1/-1->0->1 [4] 3/8/-1->0->-1 [5] 2/-1/-1->0->3 [6] 3/-1/-1->0->2 [7] -1/-1/-1->0->1 [8] 1/-1/-1->0->4 [9] -1/-1/-1->0->3 [10] 1/-1/-1->0->2 [11] 2/-1/-1->0->1 [12] 3/-1/-1->0->4 [13] 2/-1/-1->0->3 [14] 3/-1/-1->0->2 [15] -1/-1/-1->0->1
0: jzxh014:2356327:2358124 [0] NCCL INFO P2P Chunksize set to 131072
3: jzxh017:51584:53368 [2] NCCL INFO Trees [0] 15/-1/-1->14->13 [1] 15/-1/-1->14->13 [2] 12/-1/-1->14->10 [3] -1/-1/-1->14->12 [4] 13/-1/-1->14->15 [5] -1/-1/-1->14->12 [6] 12/-1/-1->14->10 [7] 13/-1/-1->14->15 [8] 15/-1/-1->14->13 [9] 15/-1/-1->14->13 [10] 12/6/-1->14->-1 [11] -1/-1/-1->14->12 [12] 13/-1/-1->14->15 [13] -1/-1/-1->14->12 [14] 12/6/-1->14->-1 [15] 13/-1/-1->14->15
3: jzxh017:51583:53369 [1] NCCL INFO P2P Chunksize set to 131072
3: jzxh017:51584:53368 [2] NCCL INFO P2P Chunksize set to 131072
1: jzxh015:2965582:2968667 [0] NCCL INFO Trees [0] 5/-1/-1->4->8 [1] -1/-1/-1->4->7 [2] 5/-1/-1->4->6 [3] 6/-1/-1->4->5 [4] 7/-1/-1->4->8 [5] 6/-1/-1->4->7 [6] 7/-1/-1->4->6 [7] -1/-1/-1->4->5 [8] 5/8/0->4->12 [9] -1/-1/-1->4->7 [10] 5/-1/-1->4->6 [11] 6/-1/-1->4->5 [12] 7/8/0->4->12 [13] 6/-1/-1->4->7 [14] 7/-1/-1->4->6 [15] -1/-1/-1->4->5
1: jzxh015:2965584:2968665 [2] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 4/-1/-1->6->10 [3] -1/-1/-1->6->4 [4] 5/-1/-1->6->7 [5] -1/-1/-1->6->4 [6] 4/-1/-1->6->10 [7] 5/-1/-1->6->7 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 4/10/2->6->14 [11] -1/-1/-1->6->4 [12] 5/-1/-1->6->7 [13] -1/-1/-1->6->4 [14] 4/10/2->6->14 [15] 5/-1/-1->6->7
2: jzxh016:896065:897839 [1] NCCL INFO comm 0x14f2fc1412c0 rank 9 nRanks 16 nNodes 4 localRanks 4 localRank 1 MNNVL 0
2: jzxh016:896067:897842 [3] NCCL INFO P2P Chunksize set to 131072
2: jzxh016:896066:897840 [2] NCCL INFO P2P Chunksize set to 131072
2: jzxh016:896064:897841 [0] NCCL INFO comm 0x15476812cf80 rank 8 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0
2: jzxh016:896065:897839 [1] NCCL INFO Trees [0] 10/-1/-1->9->8 [1] 10/5/13->9->1 [2] 11/-1/-1->9->8 [3] 8/-1/-1->9->11 [4] -1/-1/-1->9->10 [5] 11/5/13->9->1 [6] -1/-1/-1->9->11 [7] 8/-1/-1->9->10 [8] 10/-1/-1->9->8 [9] 10/-1/-1->9->5 [10] 11/-1/-1->9->8 [11] 8/-1/-1->9->11 [12] -1/-1/-1->9->10 [13] 11/-1/-1->9->5 [14] -1/-1/-1->9->11 [15] 8/-1/-1->9->10
2: jzxh016:896065:897839 [1] NCCL INFO P2P Chunksize set to 131072
1: jzxh015:2965583:2968666 [1] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->9 [2] 7/-1/-1->5->4 [3] 4/-1/-1->5->7 [4] -1/-1/-1->5->6 [5] 7/-1/-1->5->9 [6] -1/-1/-1->5->7 [7] 4/-1/-1->5->6 [8] 6/-1/-1->5->4 [9] 6/9/1->5->13 [10] 7/-1/-1->5->4 [11] 4/-1/-1->5->7 [12] -1/-1/-1->5->6 [13] 7/9/1->5->13 [14] -1/-1/-1->5->7 [15] 4/-1/-1->5->6
1: jzxh015:2965582:2968667 [0] NCCL INFO P2P Chunksize set to 131072
1: jzxh015:2965584:2968665 [2] NCCL INFO P2P Chunksize set to 131072
1: jzxh015:2965583:2968666 [1] NCCL INFO P2P Chunksize set to 131072
2: jzxh016:896064:897841 [0] NCCL INFO Trees [0] 9/4/12->8->0 [1] -1/-1/-1->8->11 [2] 9/-1/-1->8->10 [3] 10/-1/-1->8->9 [4] 11/4/12->8->0 [5] 10/-1/-1->8->11 [6] 11/-1/-1->8->10 [7] -1/-1/-1->8->9 [8] 9/-1/-1->8->4 [9] -1/-1/-1->8->11 [10] 9/-1/-1->8->10 [11] 10/-1/-1->8->9 [12] 11/-1/-1->8->4 [13] 10/-1/-1->8->11 [14] 11/-1/-1->8->10 [15] -1/-1/-1->8->9
2: jzxh016:896064:897841 [0] NCCL INFO P2P Chunksize set to 131072
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 00/0 : 13[1] -> 14[2] via P2P/CUMEM
2: jzxh016:896066:897840 [2] NCCL INFO Channel 00/0 : 10[2] -> 11[3] via P2P/CUMEM
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/CUMEM
3: jzxh017:51584:53368 [2] NCCL INFO Channel 00/0 : 14[2] -> 15[3] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 00/0 : 9[1] -> 10[2] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 03/0 : 13[1] -> 14[2] via P2P/CUMEM
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 04/0 : 13[1] -> 14[2] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 03/0 : 9[1] -> 10[2] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/CUMEM
3: jzxh017:51584:53368 [2] NCCL INFO Channel 04/0 : 14[2] -> 15[3] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 04/0 : 1[1] -> 2[2] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 07/0 : 13[1] -> 14[2] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 04/0 : 9[1] -> 10[2] via P2P/CUMEM
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/CUMEM
2: jzxh016:896066:897840 [2] NCCL INFO Channel 04/0 : 10[2] -> 11[3] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 07/0 : 1[1] -> 2[2] via P2P/CUMEM
3: jzxh017:51584:53368 [2] NCCL INFO Channel 08/0 : 14[2] -> 15[3] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 07/0 : 9[1] -> 10[2] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 08/0 : 13[1] -> 14[2] via P2P/CUMEM
2: jzxh016:896066:897840 [2] NCCL INFO Channel 08/0 : 10[2] -> 11[3] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 08/0 : 9[1] -> 10[2] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 00/0 : 5[1] -> 6[2] via P2P/CUMEM
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 00/0 : 6[2] -> 7[3] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 08/0 : 1[1] -> 2[2] via P2P/CUMEM
2: jzxh016:896066:897840 [2] NCCL INFO Channel 12/0 : 10[2] -> 11[3] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 11/0 : 13[1] -> 14[2] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 11/0 : 9[1] -> 10[2] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 11/0 : 1[1] -> 2[2] via P2P/CUMEM
3: jzxh017:51584:53368 [2] NCCL INFO Channel 12/0 : 14[2] -> 15[3] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 12/0 : 9[1] -> 10[2] via P2P/CUMEM
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 04/0 : 6[2] -> 7[3] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 03/0 : 5[1] -> 6[2] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 12/0 : 13[1] -> 14[2] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 12/0 : 1[1] -> 2[2] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 15/0 : 9[1] -> 10[2] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 15/0 : 1[1] -> 2[2] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 04/0 : 5[1] -> 6[2] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 15/0 : 13[1] -> 14[2] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 00/0 : 11[3] -> 12[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 08/0 : 6[2] -> 7[3] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 00/0 : 15[3] -> 0[0] [send] via NET/IB/0(12)/GDRDMA
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 07/0 : 5[1] -> 6[2] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 04/0 : 11[3] -> 12[0] [receive] via NET/IB/0/GDRDMA
3: jzxh017:51585:53367 [3] NCCL INFO Channel 04/0 : 15[3] -> 0[0] [send] via NET/IB/0(12)/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 12/0 : 6[2] -> 7[3] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 08/0 : 5[1] -> 6[2] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 08/0 : 11[3] -> 12[0] [receive] via NET/IB/0/GDRDMA
3: jzxh017:51585:53367 [3] NCCL INFO Channel 08/0 : 15[3] -> 0[0] [send] via NET/IB/0(12)/GDRDMA
3: jzxh017:51582:53370 [0] NCCL INFO Channel 12/0 : 11[3] -> 12[0] [receive] via NET/IB/0/GDRDMA
3: jzxh017:51582:53370 [0] NCCL INFO Channel 00/0 : 12[0] -> 13[1] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 11/0 : 5[1] -> 6[2] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 00/0 : 15[3] -> 0[0] [receive] via NET/IB/0/GDRDMA
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[0] [send] via NET/IB/0(0)/GDRDMA
3: jzxh017:51585:53367 [3] NCCL INFO Channel 12/0 : 15[3] -> 0[0] [send] via NET/IB/0(12)/GDRDMA
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 04/0 : 15[3] -> 0[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 00/0 : 3[3] -> 4[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 00/0 : 7[3] -> 8[0] [send] via NET/IB/0(4)/GDRDMA
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 12/0 : 5[1] -> 6[2] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 03/0 : 12[0] -> 13[1] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 04/0 : 3[3] -> 4[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 04/0 : 7[3] -> 8[0] [send] via NET/IB/0(4)/GDRDMA
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 04/0 : 3[3] -> 4[0] [send] via NET/IB/0(0)/GDRDMA
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 08/0 : 15[3] -> 0[0] [receive] via NET/IB/0/GDRDMA
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 08/0 : 3[3] -> 4[0] [send] via NET/IB/0(0)/GDRDMA
3: jzxh017:51582:53370 [0] NCCL INFO Channel 04/0 : 12[0] -> 13[1] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 08/0 : 3[3] -> 4[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 15/0 : 5[1] -> 6[2] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 12/0 : 15[3] -> 0[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 08/0 : 7[3] -> 8[0] [send] via NET/IB/0(4)/GDRDMA
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 12/0 : 3[3] -> 4[0] [send] via NET/IB/0(0)/GDRDMA
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 12/0 : 3[3] -> 4[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 00/0 : 4[0] -> 5[1] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 12/0 : 7[3] -> 8[0] [send] via NET/IB/0(4)/GDRDMA
3: jzxh017:51582:53370 [0] NCCL INFO Channel 07/0 : 12[0] -> 13[1] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 03/0 : 4[0] -> 5[1] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 08/0 : 12[0] -> 13[1] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 04/0 : 4[0] -> 5[1] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 11/0 : 12[0] -> 13[1] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 12/0 : 12[0] -> 13[1] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 07/0 : 4[0] -> 5[1] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 08/0 : 4[0] -> 5[1] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 15/0 : 12[0] -> 13[1] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 11/0 : 4[0] -> 5[1] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 00/0 : 7[3] -> 8[0] [receive] via NET/IB/0/GDRDMA
2: jzxh016:896067:897842 [3] NCCL INFO Channel 00/0 : 11[3] -> 12[0] [send] via NET/IB/0(8)/GDRDMA
2: jzxh016:896064:897841 [0] NCCL INFO Channel 04/0 : 7[3] -> 8[0] [receive] via NET/IB/0/GDRDMA
2: jzxh016:896067:897842 [3] NCCL INFO Channel 04/0 : 11[3] -> 12[0] [send] via NET/IB/0(8)/GDRDMA
2: jzxh016:896064:897841 [0] NCCL INFO Channel 08/0 : 7[3] -> 8[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 12/0 : 4[0] -> 5[1] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 08/0 : 11[3] -> 12[0] [send] via NET/IB/0(8)/GDRDMA
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 15/0 : 4[0] -> 5[1] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 12/0 : 7[3] -> 8[0] [receive] via NET/IB/0/GDRDMA
2: jzxh016:896064:897841 [0] NCCL INFO Channel 00/0 : 8[0] -> 9[1] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 01/0 : 12[0] -> 15[3] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 12/0 : 11[3] -> 12[0] [send] via NET/IB/0(8)/GDRDMA
2: jzxh016:896064:897841 [0] NCCL INFO Channel 03/0 : 8[0] -> 9[1] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 02/0 : 12[0] -> 15[3] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 01/0 : 4[0] -> 7[3] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 05/0 : 12[0] -> 15[3] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 04/0 : 8[0] -> 9[1] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 01/0 : 0[0] -> 3[3] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 02/0 : 4[0] -> 7[3] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 07/0 : 8[0] -> 9[1] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 02/0 : 0[0] -> 3[3] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 06/0 : 12[0] -> 15[3] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 05/0 : 4[0] -> 7[3] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 08/0 : 8[0] -> 9[1] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 06/0 : 4[0] -> 7[3] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 05/0 : 0[0] -> 3[3] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 09/0 : 12[0] -> 15[3] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 11/0 : 8[0] -> 9[1] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 09/0 : 4[0] -> 7[3] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 06/0 : 0[0] -> 3[3] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 10/0 : 12[0] -> 15[3] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 12/0 : 8[0] -> 9[1] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 10/0 : 4[0] -> 7[3] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 09/0 : 0[0] -> 3[3] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 13/0 : 12[0] -> 15[3] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 13/0 : 4[0] -> 7[3] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 15/0 : 8[0] -> 9[1] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 14/0 : 4[0] -> 7[3] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 10/0 : 0[0] -> 3[3] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 14/0 : 12[0] -> 15[3] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 13/0 : 0[0] -> 3[3] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 01/0 : 8[0] -> 11[3] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 14/0 : 0[0] -> 3[3] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 02/0 : 8[0] -> 11[3] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 05/0 : 8[0] -> 11[3] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 06/0 : 8[0] -> 11[3] via P2P/CUMEM
3: jzxh017:51584:53368 [2] NCCL INFO Channel 02/0 : 11[3] -> 14[2] [receive] via NET/IB/2/GDRDMA
3: jzxh017:51585:53367 [3] NCCL INFO Channel 02/0 : 15[3] -> 2[2] [send] via NET/IB/2(14)/GDRDMA
2: jzxh016:896064:897841 [0] NCCL INFO Channel 09/0 : 8[0] -> 11[3] via P2P/CUMEM
3: jzxh017:51584:53368 [2] NCCL INFO Channel 06/0 : 11[3] -> 14[2] [receive] via NET/IB/2/GDRDMA
3: jzxh017:51585:53367 [3] NCCL INFO Channel 06/0 : 15[3] -> 2[2] [send] via NET/IB/2(14)/GDRDMA
3: jzxh017:51584:53368 [2] NCCL INFO Channel 10/0 : 11[3] -> 14[2] [receive] via NET/IB/2/GDRDMA
3: jzxh017:51585:53367 [3] NCCL INFO Channel 10/0 : 15[3] -> 2[2] [send] via NET/IB/2(14)/GDRDMA
2: jzxh016:896064:897841 [0] NCCL INFO Channel 10/0 : 8[0] -> 11[3] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 13/0 : 8[0] -> 11[3] via P2P/CUMEM
3: jzxh017:51584:53368 [2] NCCL INFO Channel 14/0 : 11[3] -> 14[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896064:897841 [0] NCCL INFO Channel 14/0 : 8[0] -> 11[3] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 14/0 : 15[3] -> 2[2] [send] via NET/IB/2(14)/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 02/0 : 3[3] -> 6[2] [receive] via NET/IB/2/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 06/0 : 3[3] -> 6[2] [receive] via NET/IB/2/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 10/0 : 3[3] -> 6[2] [receive] via NET/IB/2/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 14/0 : 3[3] -> 6[2] [receive] via NET/IB/2/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 02/0 : 7[3] -> 10[2] [send] via NET/IB/2(6)/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 06/0 : 7[3] -> 10[2] [send] via NET/IB/2(6)/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 10/0 : 7[3] -> 10[2] [send] via NET/IB/2(6)/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 14/0 : 7[3] -> 10[2] [send] via NET/IB/2(6)/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 02/0 : 15[3] -> 2[2] [receive] via NET/IB/2/GDRDMA
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 02/0 : 3[3] -> 6[2] [send] via NET/IB/2(2)/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 06/0 : 15[3] -> 2[2] [receive] via NET/IB/2/GDRDMA
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 06/0 : 3[3] -> 6[2] [send] via NET/IB/2(2)/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 10/0 : 15[3] -> 2[2] [receive] via NET/IB/2/GDRDMA
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 10/0 : 3[3] -> 6[2] [send] via NET/IB/2(2)/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 14/0 : 15[3] -> 2[2] [receive] via NET/IB/2/GDRDMA
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 14/0 : 3[3] -> 6[2] [send] via NET/IB/2(2)/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 02/0 : 7[3] -> 10[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896067:897842 [3] NCCL INFO Channel 02/0 : 11[3] -> 14[2] [send] via NET/IB/2(10)/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 06/0 : 7[3] -> 10[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896067:897842 [3] NCCL INFO Channel 06/0 : 11[3] -> 14[2] [send] via NET/IB/2(10)/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 10/0 : 7[3] -> 10[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896067:897842 [3] NCCL INFO Channel 10/0 : 11[3] -> 14[2] [send] via NET/IB/2(10)/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 14/0 : 7[3] -> 10[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896067:897842 [3] NCCL INFO Channel 14/0 : 11[3] -> 14[2] [send] via NET/IB/2(10)/GDRDMA
3: jzxh017:51583:53369 [1] NCCL INFO Channel 01/0 : 10[2] -> 13[1] [receive] via NET/IB/1/GDRDMA
3: jzxh017:51584:53368 [2] NCCL INFO Channel 01/0 : 14[2] -> 1[1] [send] via NET/IB/1(13)/GDRDMA
3: jzxh017:51583:53369 [1] NCCL INFO Channel 05/0 : 10[2] -> 13[1] [receive] via NET/IB/1/GDRDMA
3: jzxh017:51584:53368 [2] NCCL INFO Channel 05/0 : 14[2] -> 1[1] [send] via NET/IB/1(13)/GDRDMA
3: jzxh017:51583:53369 [1] NCCL INFO Channel 09/0 : 10[2] -> 13[1] [receive] via NET/IB/1/GDRDMA
3: jzxh017:51584:53368 [2] NCCL INFO Channel 09/0 : 14[2] -> 1[1] [send] via NET/IB/1(13)/GDRDMA
3: jzxh017:51583:53369 [1] NCCL INFO Channel 13/0 : 10[2] -> 13[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 01/0 : 2[2] -> 5[1] [receive] via NET/IB/1/GDRDMA
3: jzxh017:51584:53368 [2] NCCL INFO Channel 13/0 : 14[2] -> 1[1] [send] via NET/IB/1(13)/GDRDMA
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 01/0 : 14[2] -> 1[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 01/0 : 6[2] -> 9[1] [send] via NET/IB/1(5)/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 01/0 : 2[2] -> 5[1] [send] via NET/IB/1(1)/GDRDMA
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 05/0 : 2[2] -> 5[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 05/0 : 6[2] -> 9[1] [send] via NET/IB/1(5)/GDRDMA
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 05/0 : 14[2] -> 1[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 09/0 : 2[2] -> 5[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 09/0 : 6[2] -> 9[1] [send] via NET/IB/1(5)/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 05/0 : 2[2] -> 5[1] [send] via NET/IB/1(1)/GDRDMA
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 09/0 : 14[2] -> 1[1] [receive] via NET/IB/1/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 09/0 : 2[2] -> 5[1] [send] via NET/IB/1(1)/GDRDMA
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 13/0 : 2[2] -> 5[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 13/0 : 6[2] -> 9[1] [send] via NET/IB/1(5)/GDRDMA
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 13/0 : 14[2] -> 1[1] [receive] via NET/IB/1/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 13/0 : 2[2] -> 5[1] [send] via NET/IB/1(1)/GDRDMA
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 01/0 : 5[1] -> 4[0] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 02/0 : 5[1] -> 4[0] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 01/0 : 6[2] -> 9[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 01/0 : 10[2] -> 13[1] [send] via NET/IB/1(9)/GDRDMA
2: jzxh016:896065:897839 [1] NCCL INFO Channel 05/0 : 6[2] -> 9[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 05/0 : 10[2] -> 13[1] [send] via NET/IB/1(9)/GDRDMA
2: jzxh016:896065:897839 [1] NCCL INFO Channel 09/0 : 6[2] -> 9[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 09/0 : 10[2] -> 13[1] [send] via NET/IB/1(9)/GDRDMA
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 13/0 : 6[2] -> 9[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 13/0 : 10[2] -> 13[1] [send] via NET/IB/1(9)/GDRDMA
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 01/0 : 9[1] -> 8[0] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 01/0 : 13[1] -> 12[0] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 02/0 : 9[1] -> 8[0] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 02/0 : 13[1] -> 12[0] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 05/0 : 9[1] -> 8[0] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 05/0 : 13[1] -> 12[0] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 06/0 : 9[1] -> 8[0] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 06/0 : 13[1] -> 12[0] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 09/0 : 9[1] -> 8[0] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 09/0 : 13[1] -> 12[0] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 10/0 : 13[1] -> 12[0] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 10/0 : 9[1] -> 8[0] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 13/0 : 9[1] -> 8[0] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 13/0 : 13[1] -> 12[0] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 14/0 : 13[1] -> 12[0] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 14/0 : 9[1] -> 8[0] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 03/0 : 6[2] -> 11[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 03/0 : 10[2] -> 15[3] [send] via NET/IB/3(11)/GDRDMA
2: jzxh016:896067:897842 [3] NCCL INFO Channel 07/0 : 6[2] -> 11[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 07/0 : 10[2] -> 15[3] [send] via NET/IB/3(11)/GDRDMA
2: jzxh016:896067:897842 [3] NCCL INFO Channel 11/0 : 6[2] -> 11[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 11/0 : 10[2] -> 15[3] [send] via NET/IB/3(11)/GDRDMA
2: jzxh016:896067:897842 [3] NCCL INFO Channel 15/0 : 6[2] -> 11[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 15/0 : 10[2] -> 15[3] [send] via NET/IB/3(11)/GDRDMA
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 03/0 : 14[2] -> 3[3] [receive] via NET/IB/3/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 03/0 : 2[2] -> 7[3] [send] via NET/IB/3(3)/GDRDMA
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 07/0 : 14[2] -> 3[3] [receive] via NET/IB/3/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 07/0 : 2[2] -> 7[3] [send] via NET/IB/3(3)/GDRDMA
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 11/0 : 14[2] -> 3[3] [receive] via NET/IB/3/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 11/0 : 2[2] -> 7[3] [send] via NET/IB/3(3)/GDRDMA
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 15/0 : 14[2] -> 3[3] [receive] via NET/IB/3/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 15/0 : 2[2] -> 7[3] [send] via NET/IB/3(3)/GDRDMA
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 03/0 : 3[3] -> 0[0] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 03/0 : 10[2] -> 15[3] [receive] via NET/IB/3/GDRDMA
3: jzxh017:51584:53368 [2] NCCL INFO Channel 03/0 : 14[2] -> 3[3] [send] via NET/IB/3(15)/GDRDMA
3: jzxh017:51585:53367 [3] NCCL INFO Channel 07/0 : 10[2] -> 15[3] [receive] via NET/IB/3/GDRDMA
3: jzxh017:51584:53368 [2] NCCL INFO Channel 07/0 : 14[2] -> 3[3] [send] via NET/IB/3(15)/GDRDMA
3: jzxh017:51585:53367 [3] NCCL INFO Channel 11/0 : 10[2] -> 15[3] [receive] via NET/IB/3/GDRDMA
3: jzxh017:51584:53368 [2] NCCL INFO Channel 11/0 : 14[2] -> 3[3] [send] via NET/IB/3(15)/GDRDMA
3: jzxh017:51585:53367 [3] NCCL INFO Channel 15/0 : 10[2] -> 15[3] [receive] via NET/IB/3/GDRDMA
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 07/0 : 3[3] -> 0[0] via P2P/CUMEM
3: jzxh017:51584:53368 [2] NCCL INFO Channel 15/0 : 14[2] -> 3[3] [send] via NET/IB/3(15)/GDRDMA
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 05/0 : 5[1] -> 4[0] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 03/0 : 15[3] -> 12[0] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 11/0 : 3[3] -> 0[0] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 07/0 : 15[3] -> 12[0] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 15/0 : 3[3] -> 0[0] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 11/0 : 15[3] -> 12[0] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/CUMEM
2: jzxh016:896066:897840 [2] NCCL INFO Channel 02/0 : 10[2] -> 9[1] via P2P/CUMEM
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 03/0 : 6[2] -> 11[3] [send] via NET/IB/3(7)/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 06/0 : 10[2] -> 9[1] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 15/0 : 15[3] -> 12[0] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 03/0 : 2[2] -> 7[3] [receive] via NET/IB/3/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 07/0 : 6[2] -> 11[3] [send] via NET/IB/3(7)/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 07/0 : 2[2] -> 7[3] [receive] via NET/IB/3/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 11/0 : 6[2] -> 11[3] [send] via NET/IB/3(7)/GDRDMA
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/CUMEM
2: jzxh016:896066:897840 [2] NCCL INFO Channel 10/0 : 10[2] -> 9[1] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 11/0 : 2[2] -> 7[3] [receive] via NET/IB/3/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 15/0 : 6[2] -> 11[3] [send] via NET/IB/3(7)/GDRDMA
3: jzxh017:51584:53368 [2] NCCL INFO Channel 02/0 : 14[2] -> 13[1] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 03/0 : 11[3] -> 8[0] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 15/0 : 2[2] -> 7[3] [receive] via NET/IB/3/GDRDMA
3: jzxh017:51585:53367 [3] NCCL INFO Channel 01/0 : 15[3] -> 14[2] via P2P/CUMEM
2: jzxh016:896066:897840 [2] NCCL INFO Channel 14/0 : 10[2] -> 9[1] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 03/0 : 7[3] -> 4[0] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/CUMEM
3: jzxh017:51584:53368 [2] NCCL INFO Channel 06/0 : 14[2] -> 13[1] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 07/0 : 11[3] -> 8[0] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 05/0 : 15[3] -> 14[2] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 06/0 : 5[1] -> 4[0] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 11/0 : 11[3] -> 8[0] via P2P/CUMEM
3: jzxh017:51584:53368 [2] NCCL INFO Channel 10/0 : 14[2] -> 13[1] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 09/0 : 15[3] -> 14[2] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 15/0 : 11[3] -> 8[0] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 09/0 : 5[1] -> 4[0] via P2P/CUMEM
3: jzxh017:51584:53368 [2] NCCL INFO Channel 14/0 : 14[2] -> 13[1] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 13/0 : 15[3] -> 14[2] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 01/0 : 11[3] -> 10[2] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 07/0 : 7[3] -> 4[0] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 05/0 : 11[3] -> 10[2] via P2P/CUMEM
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 02/0 : 2[2] -> 1[1] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 09/0 : 11[3] -> 10[2] via P2P/CUMEM
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 06/0 : 2[2] -> 1[1] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 13/0 : 11[3] -> 10[2] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 10/0 : 5[1] -> 4[0] via P2P/CUMEM
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 10/0 : 2[2] -> 1[1] via P2P/CUMEM
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 14/0 : 2[2] -> 1[1] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 11/0 : 7[3] -> 4[0] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 13/0 : 5[1] -> 4[0] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 15/0 : 7[3] -> 4[0] via P2P/CUMEM
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 02/0 : 6[2] -> 5[1] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 14/0 : 5[1] -> 4[0] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 01/0 : 7[3] -> 6[2] via P2P/CUMEM
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 06/0 : 6[2] -> 5[1] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 05/0 : 7[3] -> 6[2] via P2P/CUMEM
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 10/0 : 6[2] -> 5[1] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 09/0 : 7[3] -> 6[2] via P2P/CUMEM
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 14/0 : 6[2] -> 5[1] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 13/0 : 7[3] -> 6[2] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Connected all rings
3: jzxh017:51584:53368 [2] NCCL INFO Connected all rings
2: jzxh016:896066:897840 [2] NCCL INFO Connected all rings
2: jzxh016:896064:897841 [0] NCCL INFO Connected all rings
2: jzxh016:896065:897839 [1] NCCL INFO Connected all rings
2: jzxh016:896067:897842 [3] NCCL INFO Connected all rings
1: jzxh015:2965584:2968665 [2] NCCL INFO Connected all rings
1: jzxh015:2965582:2968667 [0] NCCL INFO Connected all rings
1: jzxh015:2965585:2968664 [3] NCCL INFO Connected all rings
0: jzxh014:2356329:2358125 [2] NCCL INFO Connected all rings
1: jzxh015:2965583:2968666 [1] NCCL INFO Connected all rings
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 02/0 : 4[0] -> 5[1] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Connected all rings
2: jzxh016:896064:897841 [0] NCCL INFO Channel 02/0 : 8[0] -> 9[1] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Connected all rings
3: jzxh017:51583:53369 [1] NCCL INFO Connected all rings
3: jzxh017:51582:53370 [0] NCCL INFO Channel 02/0 : 12[0] -> 13[1] via P2P/CUMEM
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 10/0 : 12[0] -> 13[1] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 10/0 : 4[0] -> 5[1] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 10/0 : 8[0] -> 9[1] via P2P/CUMEM
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/CUMEM
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 01/0 : 6[2] -> 7[3] via P2P/CUMEM
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 01/0 : 9[1] -> 10[2] via P2P/CUMEM
3: jzxh017:51584:53368 [2] NCCL INFO Channel 01/0 : 14[2] -> 15[3] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 01/0 : 13[1] -> 14[2] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 01/0 : 5[1] -> 6[2] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Connected all rings
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Connected all rings
2: jzxh016:896066:897840 [2] NCCL INFO Channel 01/0 : 10[2] -> 11[3] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 09/0 : 13[1] -> 14[2] via P2P/CUMEM
2: jzxh016:896066:897840 [2] NCCL INFO Channel 07/0 : 10[2] -> 11[3] via P2P/CUMEM
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 07/0 : 6[2] -> 7[3] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 09/0 : 9[1] -> 10[2] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 09/0 : 1[1] -> 2[2] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 02/0 : 0[0] -> 2[2] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 02/0 : 1[1] -> 3[3] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 03/0 : 1[1] -> 3[3] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 03/0 : 0[0] -> 2[2] via P2P/CUMEM
3: jzxh017:51584:53368 [2] NCCL INFO Channel 07/0 : 14[2] -> 15[3] via P2P/CUMEM
2: jzxh016:896066:897840 [2] NCCL INFO Channel 09/0 : 10[2] -> 11[3] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 09/0 : 5[1] -> 6[2] via P2P/CUMEM
3: jzxh017:51584:53368 [2] NCCL INFO Channel 09/0 : 14[2] -> 15[3] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 02/0 : 8[0] -> 10[2] via P2P/CUMEM
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 09/0 : 6[2] -> 7[3] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 02/0 : 12[0] -> 14[2] via P2P/CUMEM
3: jzxh017:51584:53368 [2] NCCL INFO Channel 15/0 : 14[2] -> 15[3] via P2P/CUMEM
2: jzxh016:896066:897840 [2] NCCL INFO Channel 15/0 : 10[2] -> 11[3] via P2P/CUMEM
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 15/0 : 6[2] -> 7[3] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 03/0 : 12[0] -> 14[2] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 03/0 : 8[0] -> 10[2] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 02/0 : 4[0] -> 6[2] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 02/0 : 13[1] -> 15[3] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 05/0 : 8[0] -> 10[2] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 02/0 : 5[1] -> 7[3] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 05/0 : 12[0] -> 14[2] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 02/0 : 9[1] -> 11[3] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 03/0 : 4[0] -> 6[2] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 06/0 : 12[0] -> 14[2] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 03/0 : 9[1] -> 11[3] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 05/0 : 4[0] -> 6[2] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 03/0 : 13[1] -> 15[3] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 06/0 : 8[0] -> 10[2] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 03/0 : 5[1] -> 7[3] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 10/0 : 8[0] -> 10[2] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 05/0 : 5[1] -> 7[3] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 05/0 : 9[1] -> 11[3] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 06/0 : 4[0] -> 6[2] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 05/0 : 1[1] -> 3[3] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 05/0 : 0[0] -> 2[2] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 06/0 : 5[1] -> 7[3] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 10/0 : 12[0] -> 14[2] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 10/0 : 4[0] -> 6[2] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 05/0 : 13[1] -> 15[3] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 06/0 : 9[1] -> 11[3] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 11/0 : 8[0] -> 10[2] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 11/0 : 12[0] -> 14[2] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 06/0 : 0[0] -> 2[2] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 06/0 : 1[1] -> 3[3] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 10/0 : 5[1] -> 7[3] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 06/0 : 13[1] -> 15[3] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 10/0 : 9[1] -> 11[3] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 11/0 : 4[0] -> 6[2] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 13/0 : 12[0] -> 14[2] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 10/0 : 1[1] -> 3[3] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 10/0 : 0[0] -> 2[2] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 13/0 : 8[0] -> 10[2] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 13/0 : 4[0] -> 6[2] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 10/0 : 13[1] -> 15[3] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 11/0 : 5[1] -> 7[3] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 14/0 : 12[0] -> 14[2] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 11/0 : 9[1] -> 11[3] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 11/0 : 13[1] -> 15[3] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 14/0 : 8[0] -> 10[2] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 11/0 : 1[1] -> 3[3] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 11/0 : 0[0] -> 2[2] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 14/0 : 4[0] -> 6[2] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 13/0 : 0[0] -> 2[2] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 13/0 : 1[1] -> 3[3] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 13/0 : 5[1] -> 7[3] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 13/0 : 13[1] -> 15[3] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 13/0 : 9[1] -> 11[3] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 14/0 : 0[0] -> 2[2] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 14/0 : 5[1] -> 7[3] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 14/0 : 13[1] -> 15[3] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 14/0 : 9[1] -> 11[3] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 04/0 : 4[0] -> 7[3] via P2P/CUMEM
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 10/0 : 2[2] -> 6[2] [receive] via NET/IB/2/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 14/0 : 2[2] -> 6[2] [receive] via NET/IB/2/GDRDMA
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 04/0 : 0[0] -> 3[3] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 04/0 : 8[0] -> 11[3] via P2P/CUMEM
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 02/0 : 6[2] -> 10[2] [send] via NET/IB/2/GDRDMA
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 12/0 : 4[0] -> 7[3] via P2P/CUMEM
2: jzxh016:896066:897840 [2] NCCL INFO Channel 02/0 : 6[2] -> 10[2] [receive] via NET/IB/2/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 06/0 : 6[2] -> 10[2] [send] via NET/IB/2/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 10/0 : 6[2] -> 10[2] [send] via NET/IB/2/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 10/0 : 2[2] -> 6[2] [send] via NET/IB/2/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 14/0 : 6[2] -> 10[2] [send] via NET/IB/2/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 14/0 : 2[2] -> 6[2] [send] via NET/IB/2/GDRDMA
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 12/0 : 0[0] -> 3[3] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 04/0 : 12[0] -> 15[3] via P2P/CUMEM
2: jzxh016:896066:897840 [2] NCCL INFO Channel 06/0 : 6[2] -> 10[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 10/0 : 6[2] -> 10[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896064:897841 [0] NCCL INFO Channel 12/0 : 8[0] -> 11[3] via P2P/CUMEM
2: jzxh016:896066:897840 [2] NCCL INFO Channel 14/0 : 6[2] -> 10[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 02/0 : 10[2] -> 14[2] [send] via NET/IB/2/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 02/0 : 10[2] -> 2[2] [receive] via NET/IB/2/GDRDMA
3: jzxh017:51584:53368 [2] NCCL INFO Channel 02/0 : 10[2] -> 14[2] [receive] via NET/IB/2/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 06/0 : 10[2] -> 2[2] [receive] via NET/IB/2/GDRDMA
3: jzxh017:51583:53369 [1] NCCL INFO Channel 01/0 : 9[1] -> 13[1] [receive] via NET/IB/1/GDRDMA
3: jzxh017:51584:53368 [2] NCCL INFO Channel 06/0 : 10[2] -> 14[2] [receive] via NET/IB/2/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 02/0 : 2[2] -> 10[2] [send] via NET/IB/2/GDRDMA
2: jzxh016:896065:897839 [1] NCCL INFO Channel 01/0 : 5[1] -> 9[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 06/0 : 10[2] -> 14[2] [send] via NET/IB/2/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 10/0 : 14[2] -> 6[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896065:897839 [1] NCCL INFO Channel 05/0 : 5[1] -> 9[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896065:897839 [1] NCCL INFO Channel 09/0 : 5[1] -> 9[1] [receive] via NET/IB/1/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 06/0 : 2[2] -> 10[2] [send] via NET/IB/2/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 14/0 : 14[2] -> 6[2] [receive] via NET/IB/2/GDRDMA
3: jzxh017:51583:53369 [1] NCCL INFO Channel 05/0 : 9[1] -> 13[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 10/0 : 6[2] -> 14[2] [send] via NET/IB/2/GDRDMA
3: jzxh017:51582:53370 [0] NCCL INFO Channel 12/0 : 12[0] -> 15[3] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 13/0 : 5[1] -> 9[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 14/0 : 6[2] -> 14[2] [send] via NET/IB/2/GDRDMA
3: jzxh017:51584:53368 [2] NCCL INFO Channel 10/0 : 6[2] -> 14[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896065:897839 [1] NCCL INFO Channel 01/0 : 9[1] -> 13[1] [send] via NET/IB/1/GDRDMA
3: jzxh017:51584:53368 [2] NCCL INFO Channel 14/0 : 6[2] -> 14[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 02/0 : 2[2] -> 10[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896065:897839 [1] NCCL INFO Channel 05/0 : 9[1] -> 13[1] [send] via NET/IB/1/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 06/0 : 2[2] -> 10[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 02/0 : 10[2] -> 2[2] [send] via NET/IB/2/GDRDMA
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 09/0 : 1[1] -> 5[1] [receive] via NET/IB/1/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 10/0 : 6[2] -> 2[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896064:897841 [0] NCCL INFO Channel 00/0 : 4[0] -> 8[0] [receive] via NET/IB/0/GDRDMA
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 14/0 : 1[1] -> 3[3] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 03/0 : 7[3] -> 11[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 06/0 : 10[2] -> 2[2] [send] via NET/IB/2/GDRDMA
3: jzxh017:51584:53368 [2] NCCL INFO Channel 10/0 : 14[2] -> 6[2] [send] via NET/IB/2/GDRDMA
2: jzxh016:896064:897841 [0] NCCL INFO Channel 04/0 : 4[0] -> 8[0] [receive] via NET/IB/0/GDRDMA
3: jzxh017:51584:53368 [2] NCCL INFO Channel 14/0 : 14[2] -> 6[2] [send] via NET/IB/2/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 14/0 : 6[2] -> 2[2] [receive] via NET/IB/2/GDRDMA
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 13/0 : 1[1] -> 5[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 01/0 : 5[1] -> 9[1] [send] via NET/IB/1/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 02/0 : 10[2] -> 6[2] [receive] via NET/IB/2/GDRDMA
3: jzxh017:51584:53368 [2] NCCL INFO Channel 02/0 : 14[2] -> 10[2] [send] via NET/IB/2/GDRDMA
2: jzxh016:896067:897842 [3] NCCL INFO Channel 07/0 : 7[3] -> 11[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896064:897841 [0] NCCL INFO Channel 08/0 : 4[0] -> 8[0] [receive] via NET/IB/0/GDRDMA
2: jzxh016:896067:897842 [3] NCCL INFO Channel 11/0 : 7[3] -> 11[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 02/0 : 14[2] -> 10[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896064:897841 [0] NCCL INFO Channel 12/0 : 4[0] -> 8[0] [receive] via NET/IB/0/GDRDMA
2: jzxh016:896067:897842 [3] NCCL INFO Channel 15/0 : 7[3] -> 11[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 06/0 : 14[2] -> 10[2] [receive] via NET/IB/2/GDRDMA
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 05/0 : 5[1] -> 9[1] [send] via NET/IB/1/GDRDMA
3: jzxh017:51584:53368 [2] NCCL INFO Channel 06/0 : 14[2] -> 10[2] [send] via NET/IB/2/GDRDMA
2: jzxh016:896064:897841 [0] NCCL INFO Channel 00/0 : 8[0] -> 12[0] [send] via NET/IB/0/GDRDMA
2: jzxh016:896067:897842 [3] NCCL INFO Channel 03/0 : 11[3] -> 15[3] [send] via NET/IB/3/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 06/0 : 10[2] -> 6[2] [receive] via NET/IB/2/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 02/0 : 10[2] -> 6[2] [send] via NET/IB/2/GDRDMA
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 09/0 : 5[1] -> 9[1] [send] via NET/IB/1/GDRDMA
2: jzxh016:896064:897841 [0] NCCL INFO Channel 04/0 : 8[0] -> 12[0] [send] via NET/IB/0/GDRDMA
2: jzxh016:896067:897842 [3] NCCL INFO Channel 07/0 : 11[3] -> 15[3] [send] via NET/IB/3/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 10/0 : 10[2] -> 6[2] [receive] via NET/IB/2/GDRDMA
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 09/0 : 1[1] -> 5[1] [send] via NET/IB/1/GDRDMA
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 13/0 : 1[1] -> 5[1] [send] via NET/IB/1/GDRDMA
3: jzxh017:51584:53368 [2] NCCL INFO Channel 02/0 : 14[2] -> 12[0] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 13/0 : 5[1] -> 9[1] [send] via NET/IB/1/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 14/0 : 10[2] -> 6[2] [receive] via NET/IB/2/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 10/0 : 6[2] -> 2[2] [send] via NET/IB/2/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 11/0 : 3[3] -> 7[3] [receive] via NET/IB/3/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 02/0 : 2[2] -> 0[0] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 08/0 : 0[0] -> 4[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 14/0 : 6[2] -> 2[2] [send] via NET/IB/2/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 06/0 : 10[2] -> 6[2] [send] via NET/IB/2/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 15/0 : 3[3] -> 7[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 10/0 : 10[2] -> 6[2] [send] via NET/IB/2/GDRDMA
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 12/0 : 0[0] -> 4[0] [receive] via NET/IB/0/GDRDMA
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 01/0 : 9[1] -> 1[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 14/0 : 10[2] -> 6[2] [send] via NET/IB/2/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 03/0 : 7[3] -> 11[3] [send] via NET/IB/3/GDRDMA
2: jzxh016:896065:897839 [1] NCCL INFO Channel 01/0 : 1[1] -> 9[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 02/0 : 10[2] -> 8[0] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 00/0 : 4[0] -> 8[0] [send] via NET/IB/0/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 07/0 : 7[3] -> 11[3] [send] via NET/IB/3/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 02/0 : 6[2] -> 4[0] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 11/0 : 7[3] -> 11[3] [send] via NET/IB/3/GDRDMA
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 04/0 : 4[0] -> 8[0] [send] via NET/IB/0/GDRDMA
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 05/0 : 9[1] -> 1[1] [receive] via NET/IB/1/GDRDMA
3: jzxh017:51583:53369 [1] NCCL INFO Channel 09/0 : 5[1] -> 13[1] [receive] via NET/IB/1/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 03/0 : 2[2] -> 0[0] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 13/0 : 5[1] -> 13[1] [receive] via NET/IB/1/GDRDMA
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 01/0 : 1[1] -> 9[1] [send] via NET/IB/1/GDRDMA
3: jzxh017:51584:53368 [2] NCCL INFO Channel 03/0 : 14[2] -> 12[0] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 09/0 : 13[1] -> 5[1] [send] via NET/IB/1/GDRDMA
3: jzxh017:51583:53369 [1] NCCL INFO Channel 13/0 : 13[1] -> 5[1] [send] via NET/IB/1/GDRDMA
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 11/0 : 3[3] -> 7[3] [send] via NET/IB/3/GDRDMA
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 05/0 : 1[1] -> 9[1] [send] via NET/IB/1/GDRDMA
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 15/0 : 3[3] -> 7[3] [send] via NET/IB/3/GDRDMA
2: jzxh016:896065:897839 [1] NCCL INFO Channel 05/0 : 1[1] -> 9[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896065:897839 [1] NCCL INFO Channel 01/0 : 9[1] -> 1[1] [send] via NET/IB/1/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 15/0 : 7[3] -> 11[3] [send] via NET/IB/3/GDRDMA
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 08/0 : 4[0] -> 8[0] [send] via NET/IB/0/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 05/0 : 2[2] -> 0[0] via P2P/CUMEM
2: jzxh016:896066:897840 [2] NCCL INFO Channel 03/0 : 10[2] -> 8[0] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 09/0 : 13[1] -> 5[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896065:897839 [1] NCCL INFO Channel 05/0 : 9[1] -> 1[1] [send] via NET/IB/1/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 03/0 : 6[2] -> 4[0] via P2P/CUMEM
2: jzxh016:896066:897840 [2] NCCL INFO Channel 05/0 : 10[2] -> 8[0] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 12/0 : 4[0] -> 8[0] [send] via NET/IB/0/GDRDMA
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 09/0 : 5[1] -> 1[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 13/0 : 13[1] -> 5[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 09/0 : 5[1] -> 13[1] [send] via NET/IB/1/GDRDMA
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 13/0 : 5[1] -> 1[1] [receive] via NET/IB/1/GDRDMA
3: jzxh017:51584:53368 [2] NCCL INFO Channel 05/0 : 14[2] -> 12[0] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 03/0 : 11[3] -> 3[3] [receive] via NET/IB/3/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 06/0 : 2[2] -> 0[0] via P2P/CUMEM
3: jzxh017:51584:53368 [2] NCCL INFO Channel 06/0 : 14[2] -> 12[0] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 01/0 : 13[1] -> 9[1] [send] via NET/IB/1/GDRDMA
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 07/0 : 11[3] -> 3[3] [receive] via NET/IB/3/GDRDMA
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 03/0 : 3[3] -> 11[3] [send] via NET/IB/3/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 06/0 : 10[2] -> 8[0] via P2P/CUMEM
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 05/0 : 6[2] -> 4[0] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 13/0 : 5[1] -> 13[1] [send] via NET/IB/1/GDRDMA
2: jzxh016:896065:897839 [1] NCCL INFO Channel 01/0 : 13[1] -> 9[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 06/0 : 6[2] -> 4[0] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 01/0 : 9[1] -> 5[1] [receive] via NET/IB/1/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 10/0 : 10[2] -> 8[0] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 05/0 : 9[1] -> 5[1] [receive] via NET/IB/1/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 10/0 : 2[2] -> 0[0] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 05/0 : 13[1] -> 9[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 11/0 : 15[3] -> 7[3] [receive] via NET/IB/3/GDRDMA
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 07/0 : 3[3] -> 11[3] [send] via NET/IB/3/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 10/0 : 6[2] -> 4[0] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 09/0 : 9[1] -> 5[1] [receive] via NET/IB/1/GDRDMA
3: jzxh017:51584:53368 [2] NCCL INFO Channel 10/0 : 14[2] -> 12[0] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 05/0 : 13[1] -> 9[1] [send] via NET/IB/1/GDRDMA
3: jzxh017:51585:53367 [3] NCCL INFO Channel 03/0 : 11[3] -> 15[3] [receive] via NET/IB/3/GDRDMA
3: jzxh017:51582:53370 [0] NCCL INFO Channel 00/0 : 8[0] -> 12[0] [receive] via NET/IB/0/GDRDMA
3: jzxh017:51584:53368 [2] NCCL INFO Channel 11/0 : 14[2] -> 12[0] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 07/0 : 11[3] -> 15[3] [receive] via NET/IB/3/GDRDMA
3: jzxh017:51582:53370 [0] NCCL INFO Channel 04/0 : 8[0] -> 12[0] [receive] via NET/IB/0/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 11/0 : 2[2] -> 0[0] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 01/0 : 9[1] -> 5[1] [send] via NET/IB/1/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 15/0 : 15[3] -> 7[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 11/0 : 10[2] -> 8[0] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 13/0 : 9[1] -> 5[1] [receive] via NET/IB/1/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 11/0 : 7[3] -> 15[3] [send] via NET/IB/3/GDRDMA
2: jzxh016:896065:897839 [1] NCCL INFO Channel 05/0 : 9[1] -> 5[1] [send] via NET/IB/1/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 15/0 : 7[3] -> 15[3] [send] via NET/IB/3/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 11/0 : 6[2] -> 4[0] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 09/0 : 5[1] -> 1[1] [send] via NET/IB/1/GDRDMA
2: jzxh016:896065:897839 [1] NCCL INFO Channel 09/0 : 9[1] -> 5[1] [send] via NET/IB/1/GDRDMA
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 13/0 : 5[1] -> 1[1] [send] via NET/IB/1/GDRDMA
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 13/0 : 2[2] -> 0[0] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 00/0 : 0[0] -> 8[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 13/0 : 6[2] -> 4[0] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 13/0 : 9[1] -> 5[1] [send] via NET/IB/1/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 13/0 : 10[2] -> 8[0] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 04/0 : 0[0] -> 8[0] [receive] via NET/IB/0/GDRDMA
3: jzxh017:51584:53368 [2] NCCL INFO Channel 13/0 : 14[2] -> 12[0] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 11/0 : 7[3] -> 15[3] [receive] via NET/IB/3/GDRDMA
3: jzxh017:51582:53370 [0] NCCL INFO Channel 08/0 : 4[0] -> 12[0] [receive] via NET/IB/0/GDRDMA
3: jzxh017:51585:53367 [3] NCCL INFO Channel 15/0 : 7[3] -> 15[3] [receive] via NET/IB/3/GDRDMA
3: jzxh017:51582:53370 [0] NCCL INFO Channel 12/0 : 4[0] -> 12[0] [receive] via NET/IB/0/GDRDMA
3: jzxh017:51584:53368 [2] NCCL INFO Channel 14/0 : 14[2] -> 12[0] via P2P/CUMEM
3: jzxh017:51582:53370 [0] NCCL INFO Channel 08/0 : 12[0] -> 4[0] [send] via NET/IB/0/GDRDMA
3: jzxh017:51585:53367 [3] NCCL INFO Channel 11/0 : 15[3] -> 7[3] [send] via NET/IB/3/GDRDMA
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 14/0 : 6[2] -> 4[0] via P2P/CUMEM
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 14/0 : 2[2] -> 0[0] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 11/0 : 7[3] -> 3[3] [receive] via NET/IB/3/GDRDMA
3: jzxh017:51582:53370 [0] NCCL INFO Channel 12/0 : 12[0] -> 4[0] [send] via NET/IB/0/GDRDMA
3: jzxh017:51585:53367 [3] NCCL INFO Channel 15/0 : 15[3] -> 7[3] [send] via NET/IB/3/GDRDMA
2: jzxh016:896067:897842 [3] NCCL INFO Channel 03/0 : 3[3] -> 11[3] [receive] via NET/IB/3/GDRDMA
3: jzxh017:51585:53367 [3] NCCL INFO Channel 03/0 : 15[3] -> 11[3] [send] via NET/IB/3/GDRDMA
2: jzxh016:896064:897841 [0] NCCL INFO Channel 00/0 : 8[0] -> 0[0] [send] via NET/IB/0/GDRDMA
2: jzxh016:896066:897840 [2] NCCL INFO Channel 14/0 : 10[2] -> 8[0] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 07/0 : 3[3] -> 11[3] [receive] via NET/IB/3/GDRDMA
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 08/0 : 0[0] -> 4[0] [send] via NET/IB/0/GDRDMA
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 15/0 : 7[3] -> 3[3] [receive] via NET/IB/3/GDRDMA
3: jzxh017:51585:53367 [3] NCCL INFO Channel 07/0 : 15[3] -> 11[3] [send] via NET/IB/3/GDRDMA
2: jzxh016:896064:897841 [0] NCCL INFO Channel 04/0 : 8[0] -> 0[0] [send] via NET/IB/0/GDRDMA
2: jzxh016:896067:897842 [3] NCCL INFO Channel 03/0 : 11[3] -> 3[3] [send] via NET/IB/3/GDRDMA
2: jzxh016:896067:897842 [3] NCCL INFO Channel 07/0 : 11[3] -> 3[3] [send] via NET/IB/3/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 03/0 : 11[3] -> 7[3] [receive] via NET/IB/3/GDRDMA
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 12/0 : 0[0] -> 4[0] [send] via NET/IB/0/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 07/0 : 11[3] -> 7[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896067:897842 [3] NCCL INFO Channel 03/0 : 15[3] -> 11[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896067:897842 [3] NCCL INFO Channel 07/0 : 15[3] -> 11[3] [receive] via NET/IB/3/GDRDMA
2: jzxh016:896067:897842 [3] NCCL INFO Channel 03/0 : 11[3] -> 7[3] [send] via NET/IB/3/GDRDMA
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 00/0 : 8[0] -> 0[0] [receive] via NET/IB/0/GDRDMA
2: jzxh016:896067:897842 [3] NCCL INFO Channel 07/0 : 11[3] -> 7[3] [send] via NET/IB/3/GDRDMA
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 04/0 : 8[0] -> 0[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 11/0 : 11[3] -> 7[3] [receive] via NET/IB/3/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 15/0 : 11[3] -> 7[3] [receive] via NET/IB/3/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 11/0 : 7[3] -> 3[3] [send] via NET/IB/3/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 15/0 : 7[3] -> 3[3] [send] via NET/IB/3/GDRDMA
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 01/0 : 3[3] -> 0[0] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 00/0 : 0[0] -> 8[0] [send] via NET/IB/0/GDRDMA
3: jzxh017:51585:53367 [3] NCCL INFO Channel 01/0 : 15[3] -> 12[0] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 11/0 : 11[3] -> 7[3] [send] via NET/IB/3/GDRDMA
2: jzxh016:896067:897842 [3] NCCL INFO Channel 15/0 : 11[3] -> 7[3] [send] via NET/IB/3/GDRDMA
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 04/0 : 0[0] -> 8[0] [send] via NET/IB/0/GDRDMA
2: jzxh016:896067:897842 [3] NCCL INFO Channel 01/0 : 11[3] -> 8[0] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 08/0 : 4[0] -> 0[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 08/0 : 12[0] -> 4[0] [receive] via NET/IB/0/GDRDMA
3: jzxh017:51585:53367 [3] NCCL INFO Channel 04/0 : 15[3] -> 12[0] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 12/0 : 12[0] -> 4[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 01/0 : 7[3] -> 4[0] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 08/0 : 4[0] -> 12[0] [send] via NET/IB/0/GDRDMA
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 12/0 : 4[0] -> 12[0] [send] via NET/IB/0/GDRDMA
2: jzxh016:896067:897842 [3] NCCL INFO Channel 04/0 : 11[3] -> 8[0] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 04/0 : 3[3] -> 0[0] via P2P/CUMEM
0: jzxh014:2356327:2358124 [0] NCCL INFO Channel 12/0 : 4[0] -> 0[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 04/0 : 7[3] -> 4[0] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 00/0 : 8[0] -> 4[0] [receive] via NET/IB/0/GDRDMA
3: jzxh017:51582:53370 [0] NCCL INFO Channel 00/0 : 12[0] -> 8[0] [send] via NET/IB/0/GDRDMA
2: jzxh016:896064:897841 [0] NCCL INFO Channel 00/0 : 12[0] -> 8[0] [receive] via NET/IB/0/GDRDMA
2: jzxh016:896064:897841 [0] NCCL INFO Channel 04/0 : 12[0] -> 8[0] [receive] via NET/IB/0/GDRDMA
2: jzxh016:896064:897841 [0] NCCL INFO Channel 00/0 : 8[0] -> 4[0] [send] via NET/IB/0/GDRDMA
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 05/0 : 3[3] -> 0[0] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 04/0 : 8[0] -> 4[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 08/0 : 8[0] -> 4[0] [receive] via NET/IB/0/GDRDMA
3: jzxh017:51582:53370 [0] NCCL INFO Channel 04/0 : 12[0] -> 8[0] [send] via NET/IB/0/GDRDMA
3: jzxh017:51585:53367 [3] NCCL INFO Channel 05/0 : 15[3] -> 12[0] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 05/0 : 11[3] -> 8[0] via P2P/CUMEM
2: jzxh016:896064:897841 [0] NCCL INFO Channel 04/0 : 8[0] -> 4[0] [send] via NET/IB/0/GDRDMA
2: jzxh016:896064:897841 [0] NCCL INFO Channel 08/0 : 8[0] -> 4[0] [send] via NET/IB/0/GDRDMA
2: jzxh016:896064:897841 [0] NCCL INFO Channel 12/0 : 8[0] -> 4[0] [send] via NET/IB/0/GDRDMA
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 12/0 : 8[0] -> 4[0] [receive] via NET/IB/0/GDRDMA
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 05/0 : 7[3] -> 4[0] via P2P/CUMEM
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 08/0 : 4[0] -> 0[0] [send] via NET/IB/0/GDRDMA
1: jzxh015:2965582:2968667 [0] NCCL INFO Channel 12/0 : 4[0] -> 0[0] [send] via NET/IB/0/GDRDMA
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 06/0 : 3[3] -> 0[0] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 06/0 : 11[3] -> 8[0] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 06/0 : 7[3] -> 4[0] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 09/0 : 3[3] -> 0[0] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 09/0 : 11[3] -> 8[0] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 09/0 : 7[3] -> 4[0] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 12/0 : 11[3] -> 8[0] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 12/0 : 7[3] -> 4[0] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 12/0 : 3[3] -> 0[0] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 13/0 : 11[3] -> 8[0] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 13/0 : 7[3] -> 4[0] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 13/0 : 3[3] -> 0[0] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 14/0 : 11[3] -> 8[0] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 14/0 : 3[3] -> 0[0] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 06/0 : 15[3] -> 12[0] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 14/0 : 7[3] -> 4[0] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 09/0 : 15[3] -> 12[0] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 12/0 : 15[3] -> 12[0] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 13/0 : 15[3] -> 12[0] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 14/0 : 15[3] -> 12[0] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 02/0 : 11[3] -> 9[1] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 02/0 : 15[3] -> 13[1] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 03/0 : 11[3] -> 9[1] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 03/0 : 15[3] -> 13[1] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 05/0 : 11[3] -> 9[1] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 02/0 : 3[3] -> 1[1] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 05/0 : 15[3] -> 13[1] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 02/0 : 7[3] -> 5[1] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 06/0 : 11[3] -> 9[1] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 03/0 : 3[3] -> 1[1] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 06/0 : 15[3] -> 13[1] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 10/0 : 11[3] -> 9[1] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 03/0 : 7[3] -> 5[1] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 05/0 : 3[3] -> 1[1] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 10/0 : 15[3] -> 13[1] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 11/0 : 11[3] -> 9[1] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 05/0 : 7[3] -> 5[1] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 06/0 : 3[3] -> 1[1] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 11/0 : 15[3] -> 13[1] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 06/0 : 7[3] -> 5[1] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 13/0 : 11[3] -> 9[1] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 10/0 : 3[3] -> 1[1] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 13/0 : 15[3] -> 13[1] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 10/0 : 7[3] -> 5[1] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 11/0 : 3[3] -> 1[1] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 14/0 : 15[3] -> 13[1] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 11/0 : 7[3] -> 5[1] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 13/0 : 3[3] -> 1[1] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 00/0 : 15[3] -> 14[2] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 13/0 : 7[3] -> 5[1] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 14/0 : 3[3] -> 1[1] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 14/0 : 7[3] -> 5[1] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 04/0 : 15[3] -> 14[2] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 00/0 : 7[3] -> 6[2] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 07/0 : 15[3] -> 14[2] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 04/0 : 7[3] -> 6[2] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 07/0 : 7[3] -> 6[2] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 00/0 : 13[1] -> 12[0] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 08/0 : 7[3] -> 6[2] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 08/0 : 15[3] -> 14[2] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 12/0 : 7[3] -> 6[2] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 03/0 : 13[1] -> 12[0] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 12/0 : 15[3] -> 14[2] via P2P/CUMEM
3: jzxh017:51585:53367 [3] NCCL INFO Channel 15/0 : 15[3] -> 14[2] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 14/0 : 11[3] -> 9[1] via P2P/CUMEM
1: jzxh015:2965585:2968664 [3] NCCL INFO Channel 15/0 : 7[3] -> 6[2] via P2P/CUMEM
2: jzxh016:896066:897840 [2] NCCL INFO Channel 00/0 : 10[2] -> 9[1] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 00/0 : 11[3] -> 10[2] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/CUMEM
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/CUMEM
3: jzxh017:51584:53368 [2] NCCL INFO Channel 00/0 : 14[2] -> 13[1] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 04/0 : 11[3] -> 10[2] via P2P/CUMEM
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 00/0 : 6[2] -> 5[1] via P2P/CUMEM
2: jzxh016:896066:897840 [2] NCCL INFO Channel 01/0 : 10[2] -> 9[1] via P2P/CUMEM
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/CUMEM
3: jzxh017:51584:53368 [2] NCCL INFO Channel 01/0 : 14[2] -> 13[1] via P2P/CUMEM
2: jzxh016:896066:897840 [2] NCCL INFO Channel 04/0 : 10[2] -> 9[1] via P2P/CUMEM
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 01/0 : 6[2] -> 5[1] via P2P/CUMEM
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 04/0 : 2[2] -> 1[1] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 07/0 : 13[1] -> 12[0] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 07/0 : 11[3] -> 10[2] via P2P/CUMEM
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 04/0 : 6[2] -> 5[1] via P2P/CUMEM
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 07/0 : 2[2] -> 1[1] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/CUMEM
3: jzxh017:51584:53368 [2] NCCL INFO Channel 04/0 : 14[2] -> 13[1] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 08/0 : 11[3] -> 10[2] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 00/0 : 5[1] -> 4[0] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 08/0 : 13[1] -> 12[0] via P2P/CUMEM
2: jzxh016:896066:897840 [2] NCCL INFO Channel 07/0 : 10[2] -> 9[1] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/CUMEM
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 08/0 : 2[2] -> 1[1] via P2P/CUMEM
2: jzxh016:896066:897840 [2] NCCL INFO Channel 08/0 : 10[2] -> 9[1] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 12/0 : 11[3] -> 10[2] via P2P/CUMEM
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 07/0 : 6[2] -> 5[1] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 03/0 : 5[1] -> 4[0] via P2P/CUMEM
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 08/0 : 6[2] -> 5[1] via P2P/CUMEM
3: jzxh017:51584:53368 [2] NCCL INFO Channel 07/0 : 14[2] -> 13[1] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 11/0 : 13[1] -> 12[0] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 07/0 : 5[1] -> 4[0] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/CUMEM
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 09/0 : 2[2] -> 1[1] via P2P/CUMEM
3: jzxh017:51583:53369 [1] NCCL INFO Channel 15/0 : 13[1] -> 12[0] via P2P/CUMEM
3: jzxh017:51584:53368 [2] NCCL INFO Channel 08/0 : 14[2] -> 13[1] via P2P/CUMEM
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 09/0 : 6[2] -> 5[1] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 08/0 : 5[1] -> 4[0] via P2P/CUMEM
2: jzxh016:896066:897840 [2] NCCL INFO Channel 09/0 : 10[2] -> 9[1] via P2P/CUMEM
3: jzxh017:51584:53368 [2] NCCL INFO Channel 09/0 : 14[2] -> 13[1] via P2P/CUMEM
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 12/0 : 2[2] -> 1[1] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/CUMEM
2: jzxh016:896067:897842 [3] NCCL INFO Channel 15/0 : 11[3] -> 10[2] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 11/0 : 5[1] -> 4[0] via P2P/CUMEM
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 12/0 : 6[2] -> 5[1] via P2P/CUMEM
3: jzxh017:51584:53368 [2] NCCL INFO Channel 12/0 : 14[2] -> 13[1] via P2P/CUMEM
2: jzxh016:896066:897840 [2] NCCL INFO Channel 12/0 : 10[2] -> 9[1] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/CUMEM
0: jzxh014:2356329:2358125 [2] NCCL INFO Channel 15/0 : 2[2] -> 1[1] via P2P/CUMEM
1: jzxh015:2965584:2968665 [2] NCCL INFO Channel 15/0 : 6[2] -> 5[1] via P2P/CUMEM
2: jzxh016:896066:897840 [2] NCCL INFO Channel 15/0 : 10[2] -> 9[1] via P2P/CUMEM
1: jzxh015:2965583:2968666 [1] NCCL INFO Channel 15/0 : 5[1] -> 4[0] via P2P/CUMEM
0: jzxh014:2356328:2358126 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 00/0 : 9[1] -> 8[0] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 03/0 : 9[1] -> 8[0] via P2P/CUMEM
3: jzxh017:51584:53368 [2] NCCL INFO Channel 15/0 : 14[2] -> 13[1] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 07/0 : 9[1] -> 8[0] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 08/0 : 9[1] -> 8[0] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 11/0 : 9[1] -> 8[0] via P2P/CUMEM
2: jzxh016:896065:897839 [1] NCCL INFO Channel 15/0 : 9[1] -> 8[0] via P2P/CUMEM
0: jzxh014:2356330:2358127 [3] NCCL INFO Connected all trees
0: jzxh014:2356327:2358124 [0] NCCL INFO Connected all trees
3: jzxh017:51582:53370 [0] NCCL INFO Connected all trees
2: jzxh016:896064:897841 [0] NCCL INFO Connected all trees
2: jzxh016:896067:897842 [3] NCCL INFO Connected all trees
1: jzxh015:2965582:2968667 [0] NCCL INFO Connected all trees
0: jzxh014:2356330:2358127 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
0: jzxh014:2356330:2358127 [3] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
3: jzxh017:51582:53370 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
2: jzxh016:896067:897842 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
2: jzxh016:896066:897840 [2] NCCL INFO Connected all trees
2: jzxh016:896064:897841 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
2: jzxh016:896067:897842 [3] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
2: jzxh016:896064:897841 [0] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
2: jzxh016:896065:897839 [1] NCCL INFO Connected all trees
1: jzxh015:2965582:2968667 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
1: jzxh015:2965582:2968667 [0] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
0: jzxh014:2356329:2358125 [2] NCCL INFO Connected all trees
0: jzxh014:2356327:2358124 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
0: jzxh014:2356327:2358124 [0] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
0: jzxh014:2356328:2358126 [1] NCCL INFO Connected all trees
0: jzxh014:2356329:2358125 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
3: jzxh017:51582:53370 [0] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
3: jzxh017:51585:53367 [3] NCCL INFO Connected all trees
2: jzxh016:896066:897840 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
2: jzxh016:896066:897840 [2] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
2: jzxh016:896065:897839 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
1: jzxh015:2965585:2968664 [3] NCCL INFO Connected all trees
0: jzxh014:2356329:2358125 [2] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
0: jzxh014:2356328:2358126 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
0: jzxh014:2356328:2358126 [1] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
3: jzxh017:51585:53367 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
3: jzxh017:51583:53369 [1] NCCL INFO Connected all trees
3: jzxh017:51585:53367 [3] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
3: jzxh017:51584:53368 [2] NCCL INFO Connected all trees
2: jzxh016:896065:897839 [1] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
1: jzxh015:2965583:2968666 [1] NCCL INFO Connected all trees
3: jzxh017:51583:53369 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
3: jzxh017:51583:53369 [1] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
3: jzxh017:51584:53368 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
1: jzxh015:2965585:2968664 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
1: jzxh015:2965585:2968664 [3] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
3: jzxh017:51584:53368 [2] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
1: jzxh015:2965583:2968666 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
1: jzxh015:2965583:2968666 [1] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
1: jzxh015:2965584:2968665 [2] NCCL INFO Connected all trees
1: jzxh015:2965584:2968665 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
1: jzxh015:2965584:2968665 [2] NCCL INFO 16 coll channels, 16 collnet channels, 0 nvls channels, 16 p2p channels, 2 p2p channels per peer
3: jzxh017:51582:53370 [0] NCCL INFO ncclCommInitRank comm 0x14552c11e540 rank 12 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x619ce177681b4aa5 - Init COMPLETE
3: jzxh017:51584:53368 [2] NCCL INFO ncclCommInitRank comm 0x14ca90136dc0 rank 14 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x619ce177681b4aa5 - Init COMPLETE
2: jzxh016:896067:897842 [3] NCCL INFO ncclCommInitRank comm 0x14a628114540 rank 11 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x619ce177681b4aa5 - Init COMPLETE
2: jzxh016:896065:897839 [1] NCCL INFO ncclCommInitRank comm 0x14f2fc1412c0 rank 9 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x619ce177681b4aa5 - Init COMPLETE
2: jzxh016:896066:897840 [2] NCCL INFO ncclCommInitRank comm 0x1498d4139700 rank 10 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x619ce177681b4aa5 - Init COMPLETE
2: jzxh016:896064:897841 [0] NCCL INFO ncclCommInitRank comm 0x15476812cf80 rank 8 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x619ce177681b4aa5 - Init COMPLETE
3: jzxh017:51585:53367 [3] NCCL INFO ncclCommInitRank comm 0x1459cc121240 rank 15 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x619ce177681b4aa5 - Init COMPLETE
3: jzxh017:51583:53369 [1] NCCL INFO ncclCommInitRank comm 0x14b65c1450c0 rank 13 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x619ce177681b4aa5 - Init COMPLETE
1: jzxh015:2965585:2968664 [3] NCCL INFO ncclCommInitRank comm 0x145b10135ed0 rank 7 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x619ce177681b4aa5 - Init COMPLETE
1: jzxh015:2965583:2968666 [1] NCCL INFO ncclCommInitRank comm 0x14e36c136200 rank 5 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x619ce177681b4aa5 - Init COMPLETE
1: jzxh015:2965582:2968667 [0] NCCL INFO ncclCommInitRank comm 0x151e3812df00 rank 4 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x619ce177681b4aa5 - Init COMPLETE
1: jzxh015:2965584:2968665 [2] NCCL INFO ncclCommInitRank comm 0x149484114480 rank 6 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x619ce177681b4aa5 - Init COMPLETE
0: jzxh014:2356329:2358125 [2] NCCL INFO ncclCommInitRank comm 0x153460122200 rank 2 nranks 16 cudaDev 2 nvmlDev 2 busId 9d000 commId 0x619ce177681b4aa5 - Init COMPLETE
0: jzxh014:2356327:2358124 [0] NCCL INFO ncclCommInitRank comm 0x1472b8130d20 rank 0 nranks 16 cudaDev 0 nvmlDev 0 busId 1b000 commId 0x619ce177681b4aa5 - Init COMPLETE
0: jzxh014:2356330:2358127 [3] NCCL INFO ncclCommInitRank comm 0x14c18412ee00 rank 3 nranks 16 cudaDev 3 nvmlDev 3 busId ad000 commId 0x619ce177681b4aa5 - Init COMPLETE
0: jzxh014:2356328:2358126 [1] NCCL INFO ncclCommInitRank comm 0x15475412d200 rank 1 nranks 16 cudaDev 1 nvmlDev 1 busId 2c000 commId 0x619ce177681b4aa5 - Init COMPLETE
0: {'loss': 0.744, 'grad_norm': 2.3311184266583216, 'learning_rate': 9.05e-07, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.0}
0: 
  0%|          | 0/2984 [00:00<?, ?it/s]
  0%|          | 1/2984 [00:43<36:11:43, 43.68s/it]
  0%|          | 2/2984 [00:57<21:26:28, 25.88s/it]
  0%|          | 3/2984 [01:08<15:48:20, 19.09s/it]
  0%|          | 4/2984 [01:18<13:06:41, 15.84s/it]
  0%|          | 5/2984 [01:29<11:36:30, 14.03s/it]
  0%|          | 6/2984 [01:40<10:43:07, 12.96s/it]
  0%|          | 7/2984 [01:51<10:08:29, 12.26s/it]
  0%|          | 8/2984 [02:02<9:46:59, 11.83s/it] 
  0%|          | 9/2984 [02:13<9:36:10, 11.62s/it]
  0%|          | 10/2984 [02:24<9:28:10, 11.46s/it]
                                                   

  0%|          | 10/2984 [02:24<9:28:10, 11.46s/it]
  0%|          | 11/2984 [02:35<9:21:32, 11.33s/it]
  0%|          | 12/2984 [02:46<9:17:46, 11.26s/it]
  0%|          | 13/2984 [02:57<9:14:53, 11.21s/it]
  0%|          | 14/2984 [03:09<9:14:17, 11.20s/it]
  1%|          | 15/2984 [03:20<9:12:36, 11.17s/it]
  1%|          | 16/2984 [03:31<9:11:52, 11.16s/it]
  1%|          | 17/2984 [03:42<9:10:26, 11.13
0: {'loss': 0.6767, 'grad_norm': 1.433371383105009, 'learning_rate': 1.3550000000000002e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.0}
0: {'loss': 0.6085, 'grad_norm': 0.815709326035465, 'learning_rate': 1.805e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.01}
0: s/it]
  1%|          | 18/2984 [03:53<9:09:05, 11.11s/it]
  1%|          | 19/2984 [04:04<9:07:54, 11.09s/it]
  1%|          | 20/2984 [04:15<9:07:11, 11.08s/it]
                                                   

  1%|          | 20/2984 [04:15<9:07:11, 11.08s/it]
  1%|          | 21/2984 [04:26<9:05:44, 11.05s/it]
  1%|          | 22/2984 [04:37<9:04:01, 11.02s/it]
  1%|          | 23/2984 [04:48<9:03:24, 11.01s/it]
  1%|          | 24/2984 [04:59<9:03:59, 11.03s/it]
  1%|          | 25/2984 [05:10<9:03:39, 11.02s/it]
  1%|          | 26/2984 [05:21<9:03:37, 11.03s/it]
  1%|          | 27/2984 [05:32<9:03:46, 11.03s/it]
  1%|          | 28/2984 [05:43<9:03:13, 11.03s/it]
  1%|          | 29/2984 [05:54<9:03:01, 11.03s/it]
  1%|          | 30/2984 [06:05<9:00:10, 10.97s/it]
                                                   

  1%|          | 30/2984 [06:05<9:00:10, 10.97s/it]
  1%|          | 31/2984 [06:16<8:59:21, 10.96s/it]
  1%|          | 32/2984 [06:27<9:01:44, 11.01s/it]
  1%|          | 33/2984 [06
0: {'loss': 0.599, 'grad_norm': 0.7657530325603907, 'learning_rate': 2.2550000000000004e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.01}
0: :38<9:02:10, 11.02s/it]
  1%|          | 34/2984 [06:49<9:02:07, 11.03s/it]
  1%|          | 35/2984 [07:00<9:02:13, 11.03s/it]
  1%|          | 36/2984 [07:12<9:07:24, 11.14s/it]
  1%|          | 37/2984 [07:23<9:05:41, 11.11s/it]
  1%|▏         | 38/2984 [07:33<9:01:57, 11.04s/it]
  1%|▏         | 39/2984 [07:44<8:59:58, 11.00s/it]
  1%|▏         | 40/2984 [07:55<8:59:08, 10.99s/it]
                                                   

  1%|▏         | 40/2984 [07:55<8:59:08, 10.99s/it]
  1%|▏         | 41/2984 [08:06<9:00:28, 11.02s/it]
  1%|▏         | 42/2984 [08:17<8:59:43, 11.01s/it]
  1%|▏         | 43/2984 [08:28<8:58:39, 10.99s/it]
  1%|▏         | 44/2984 [08:39<8:57:25, 10.97s/it]
  2%|▏         | 45/2984 [08:50<8:56:07, 10.95s/it]
  2%|▏         | 46/2984 [09:01<8:55:25, 10.93s/it]
  2%|▏         | 47/2984 [09:12<8:54:09, 10.91s/it]
  2%|▏         | 48/2984 [09:23<8:52:55, 10.89s/it]
  2%|▏         | 49/2984 [09:34<8:52:19, 10.88s/it]
  2%|▏         | 50/2984 [09:45<8:5
0: {'loss': 0.582, 'grad_norm': 0.723108681396044, 'learning_rate': 2.7050000000000004e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.01}
0: {'loss': 0.5741, 'grad_norm': 0.7705556734660447, 'learning_rate': 3.1550000000000003e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.01}
0: 2:37, 10.89s/it]
                                                   

  2%|▏         | 50/2984 [09:45<8:52:37, 10.89s/it]
  2%|▏         | 51/2984 [09:55<8:52:01, 10.88s/it]
  2%|▏         | 52/2984 [10:06<8:51:51, 10.88s/it]
  2%|▏         | 53/2984 [10:17<8:51:51, 10.89s/it]
  2%|▏         | 54/2984 [10:28<8:49:06, 10.83s/it]
  2%|▏         | 55/2984 [10:39<8:45:39, 10.77s/it]
  2%|▏         | 56/2984 [10:49<8:43:49, 10.73s/it]
  2%|▏         | 57/2984 [11:00<8:41:58, 10.70s/it]
  2%|▏         | 58/2984 [11:10<8:40:32, 10.67s/it]
  2%|▏         | 59/2984 [11:21<8:39:41, 10.66s/it]
  2%|▏         | 60/2984 [11:32<8:39:35, 10.66s/it]
                                                   

  2%|▏         | 60/2984 [11:32<8:39:35, 10.66s/it]
  2%|▏         | 61/2984 [11:42<8:39:55, 10.67s/it]
  2%|▏         | 62/2984 [11:53<8:40:04, 10.68s/it]
  2%|▏         | 63/2984 [12:04<8:39:24, 10.67s/it]
  2%|▏         | 64/2984 [12:14<8:39:02, 10.67s/it]
  2%|▏         | 65/2984 [12:25<8:4
0: {'loss': 0.5628, 'grad_norm': 0.7408120786031377, 'learning_rate': 3.6050000000000002e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.01}
0: {'loss': 0.5533, 'grad_norm': 0.8217468766927277, 'learning_rate': 4.055000000000001e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.02}
0: 3:00, 10.75s/it]
  2%|▏         | 66/2984 [12:36<8:44:27, 10.78s/it]
  2%|▏         | 67/2984 [12:47<8:46:02, 10.82s/it]
  2%|▏         | 68/2984 [12:58<8:44:11, 10.79s/it]
  2%|▏         | 69/2984 [13:09<8:43:08, 10.77s/it]
  2%|▏         | 70/2984 [13:19<8:41:45, 10.74s/it]
                                                   

  2%|▏         | 70/2984 [13:19<8:41:45, 10.74s/it]
  2%|▏         | 71/2984 [13:30<8:43:11, 10.78s/it]
  2%|▏         | 72/2984 [13:41<8:38:46, 10.69s/it]
  2%|▏         | 73/2984 [13:51<8:34:07, 10.60s/it]
  2%|▏         | 74/2984 [14:01<8:31:35, 10.55s/it]
  3%|▎         | 75/2984 [14:12<8:27:52, 10.48s/it]
  3%|▎         | 76/2984 [14:22<8:25:05, 10.42s/it]
  3%|▎         | 77/2984 [14:32<8:21:09, 10.34s/it]
  3%|▎         | 78/2984 [14:42<8:19:09, 10.31s/it]
  3%|▎         | 79/2984 [14:53<8:19:22, 10.31s/it]
  3%|▎         | 80/2984 [15:03<8:18:17, 10.30s/it]
                                                   

  3%|▎         | 80/2984 [15:03<8:1
0: {'loss': 0.5353, 'grad_norm': 0.794624531611395, 'learning_rate': 4.505e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.02}
0: 8:17, 10.30s/it]
  3%|▎         | 81/2984 [15:14<8:21:44, 10.37s/it]
  3%|▎         | 82/2984 [15:24<8:25:08, 10.44s/it]
  3%|▎         | 83/2984 [15:35<8:26:02, 10.47s/it]
  3%|▎         | 84/2984 [15:45<8:27:01, 10.49s/it]
  3%|▎         | 85/2984 [15:56<8:27:04, 10.49s/it]
  3%|▎         | 86/2984 [16:06<8:27:24, 10.51s/it]
  3%|▎         | 87/2984 [16:17<8:30:07, 10.57s/it]
  3%|▎         | 88/2984 [16:28<8:30:37, 10.58s/it]
  3%|▎         | 89/2984 [16:38<8:30:25, 10.58s/it]
  3%|▎         | 90/2984 [16:49<8:30:48, 10.59s/it]
                                                   

  3%|▎         | 90/2984 [16:49<8:30:48, 10.59s/it]
  3%|▎         | 91/2984 [16:59<8:30:27, 10.59s/it]
  3%|▎         | 92/2984 [17:10<8:31:37, 10.61s/it]
  3%|▎         | 93/2984 [17:21<8:30:30, 10.60s/it]
  3%|▎         | 94/2984 [17:31<8:29:15, 10.57s/it]
  3%|▎         | 95/2984 [17:42<8:28:53, 10.57s/it]
  3%|▎         | 96/2984 [17:52<8:31:34, 10.63s/it]
  3%|▎         | 97/2984 [18:03<8:
0: {'loss': 0.535, 'grad_norm': 0.8174201545451896, 'learning_rate': 4.955e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.02}
0: {'loss': 0.5399, 'grad_norm': 0.781349245376017, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.02}
0: 31:54, 10.64s/it]
  3%|▎         | 98/2984 [18:14<8:29:59, 10.60s/it]
  3%|▎         | 99/2984 [18:24<8:29:03, 10.59s/it]
  3%|▎         | 100/2984 [18:35<8:29:04, 10.59s/it]
                                                    

  3%|▎         | 100/2984 [18:35<8:29:04, 10.59s/it]
  3%|▎         | 101/2984 [18:45<8:28:53, 10.59s/it]
  3%|▎         | 102/2984 [18:56<8:28:57, 10.60s/it]
  3%|▎         | 103/2984 [19:07<8:28:40, 10.59s/it]
  3%|▎         | 104/2984 [19:17<8:27:05, 10.56s/it]
  4%|▎         | 105/2984 [19:27<8:24:37, 10.52s/it]
  4%|▎         | 106/2984 [19:38<8:24:56, 10.53s/it]
  4%|▎         | 107/2984 [19:49<8:25:28, 10.54s/it]
  4%|▎         | 108/2984 [19:59<8:25:17, 10.54s/it]
  4%|▎         | 109/2984 [20:10<8:24:53, 10.54s/it]
  4%|▎         | 110/2984 [20:20<8:25:35, 10.56s/it]
                                                    

  4%|▎         | 110/2984 [20:20<8:25:35, 10.56s/it]
  4%|▎         | 111/2984 [20:31<8:29:01, 10.63s/it]
  4%|▍         | 1
0: {'loss': 0.5303, 'grad_norm': 0.7449397635830665, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.02}
0: 12/2984 [20:42<8:29:01, 10.63s/it]
  4%|▍         | 113/2984 [20:52<8:28:56, 10.64s/it]
  4%|▍         | 114/2984 [21:03<8:30:37, 10.68s/it]
  4%|▍         | 115/2984 [21:14<8:31:55, 10.71s/it]
  4%|▍         | 116/2984 [21:24<8:30:23, 10.68s/it]
  4%|▍         | 117/2984 [21:35<8:28:19, 10.64s/it]
  4%|▍         | 118/2984 [21:46<8:27:20, 10.62s/it]
  4%|▍         | 119/2984 [21:56<8:26:11, 10.60s/it]
  4%|▍         | 120/2984 [22:07<8:24:46, 10.58s/it]
                                                    

  4%|▍         | 120/2984 [22:07<8:24:46, 10.58s/it]
  4%|▍         | 121/2984 [22:17<8:24:35, 10.57s/it]
  4%|▍         | 122/2984 [22:28<8:23:29, 10.56s/it]
  4%|▍         | 123/2984 [22:38<8:22:18, 10.53s/it]
  4%|▍         | 124/2984 [22:49<8:22:18, 10.54s/it]
  4%|▍         | 125/2984 [22:59<8:23:50, 10.57s/it]
  4%|▍         | 126/2984 [23:10<8:24:30, 10.59s/it]
  4%|▍         | 127/2984 [23:21<8:22:58, 10.56s/it]
  4%|▍         | 128/2984 [23:31<8:21:17, 10.53s/it]
0: {'loss': 0.5184, 'grad_norm': 0.840260631239892, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.03}
0: {'loss': 0.5146, 'grad_norm': 0.8046976957295178, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.03}
0:   4%|▍         | 129/2984 [23:41<8:18:08, 10.47s/it]
  4%|▍         | 130/2984 [23:52<8:16:30, 10.44s/it]
                                                    

  4%|▍         | 130/2984 [23:52<8:16:30, 10.44s/it]
  4%|▍         | 131/2984 [24:02<8:15:50, 10.43s/it]
  4%|▍         | 132/2984 [24:13<8:15:51, 10.43s/it]
  4%|▍         | 133/2984 [24:23<8:14:28, 10.41s/it]
  4%|▍         | 134/2984 [24:33<8:14:16, 10.41s/it]
  5%|▍         | 135/2984 [24:44<8:15:01, 10.43s/it]
  5%|▍         | 136/2984 [24:54<8:18:10, 10.50s/it]
  5%|▍         | 137/2984 [25:05<8:17:06, 10.48s/it]
  5%|▍         | 138/2984 [25:15<8:16:23, 10.47s/it]
  5%|▍         | 139/2984 [25:26<8:15:59, 10.46s/it]
  5%|▍         | 140/2984 [25:36<8:15:46, 10.46s/it]
                                                    

  5%|▍         | 140/2984 [25:36<8:15:46, 10.46s/it]
  5%|▍         | 141/2984 [25:47<8:14:46, 10.44s/it]
  5%|▍         | 142/2984 [25:57<8:12:59, 10.41s/it]
  5%|▍         | 143/2984 [26:07<8
0: {'loss': 0.5179, 'grad_norm': 0.8160836055516464, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.03}
0: :14:03, 10.43s/it]
  5%|▍         | 144/2984 [26:18<8:14:00, 10.44s/it]
  5%|▍         | 145/2984 [26:28<8:13:40, 10.43s/it]
  5%|▍         | 146/2984 [26:39<8:13:10, 10.43s/it]
  5%|▍         | 147/2984 [26:49<8:13:24, 10.43s/it]
  5%|▍         | 148/2984 [27:00<8:13:28, 10.44s/it]
  5%|▍         | 149/2984 [27:10<8:13:25, 10.44s/it]
  5%|▌         | 150/2984 [27:21<8:13:39, 10.45s/it]
                                                    

  5%|▌         | 150/2984 [27:21<8:13:39, 10.45s/it]
  5%|▌         | 151/2984 [27:31<8:13:47, 10.46s/it]
  5%|▌         | 152/2984 [27:42<8:17:35, 10.54s/it]
  5%|▌         | 153/2984 [27:52<8:17:07, 10.54s/it]
  5%|▌         | 154/2984 [28:03<8:16:38, 10.53s/it]
  5%|▌         | 155/2984 [28:13<8:15:53, 10.52s/it]
  5%|▌         | 156/2984 [28:24<8:16:48, 10.54s/it]
  5%|▌         | 157/2984 [28:34<8:17:10, 10.55s/it]
  5%|▌         | 158/2984 [28:45<8:14:57, 10.51s/it]
  5%|▌         | 159/2984 [28:55<8:14:49, 10.51s/it]
  5%|▌        
0: {'loss': 0.5154, 'grad_norm': 0.7536503565597004, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.03}
0: {'loss': 0.5092, 'grad_norm': 0.7723484775412541, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.03}
0:  | 160/2984 [29:06<8:14:14, 10.50s/it]
                                                    

  5%|▌         | 160/2984 [29:06<8:14:14, 10.50s/it]
  5%|▌         | 161/2984 [29:16<8:16:08, 10.55s/it]
  5%|▌         | 162/2984 [29:27<8:14:17, 10.51s/it]
  5%|▌         | 163/2984 [29:37<8:12:30, 10.48s/it]
  5%|▌         | 164/2984 [29:48<8:12:01, 10.47s/it]
  6%|▌         | 165/2984 [29:58<8:10:32, 10.44s/it]
  6%|▌         | 166/2984 [30:08<8:08:35, 10.40s/it]
  6%|▌         | 167/2984 [30:19<8:06:35, 10.36s/it]
  6%|▌         | 168/2984 [30:29<8:05:18, 10.34s/it]
  6%|▌         | 169/2984 [30:39<8:05:08, 10.34s/it]
  6%|▌         | 170/2984 [30:50<8:04:55, 10.34s/it]
                                                    

  6%|▌         | 170/2984 [30:50<8:04:55, 10.34s/it]
  6%|▌         | 171/2984 [31:00<8:06:14, 10.37s/it]
  6%|▌         | 172/2984 [31:11<8:07:35, 10.40s/it]
  6%|▌         | 173/2984 [31:21<8:10:14, 10.46s/it]
  6%|▌         | 174/2984 [31:32<8:11:51, 10.50s/i
0: {'loss': 0.4962, 'grad_norm': 0.7517380794941464, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.04}
0: t]
  6%|▌         | 175/2984 [31:42<8:11:00, 10.49s/it]
  6%|▌         | 176/2984 [31:53<8:10:40, 10.48s/it]
  6%|▌         | 177/2984 [32:03<8:10:39, 10.49s/it]
  6%|▌         | 178/2984 [32:14<8:08:47, 10.45s/it]
  6%|▌         | 179/2984 [32:24<8:07:56, 10.44s/it]
  6%|▌         | 180/2984 [32:34<8:07:07, 10.42s/it]
                                                    

  6%|▌         | 180/2984 [32:34<8:07:07, 10.42s/it]
  6%|▌         | 181/2984 [32:45<8:07:55, 10.44s/it]
  6%|▌         | 182/2984 [32:55<8:07:13, 10.43s/it]
  6%|▌         | 183/2984 [33:06<8:06:40, 10.42s/it]
  6%|▌         | 184/2984 [33:16<8:05:51, 10.41s/it]
  6%|▌         | 185/2984 [33:27<8:07:02, 10.44s/it]
  6%|▌         | 186/2984 [33:37<8:07:41, 10.46s/it]
  6%|▋         | 187/2984 [33:47<8:05:45, 10.42s/it]
  6%|▋         | 188/2984 [33:58<8:03:19, 10.37s/it]
  6%|▋         | 189/2984 [34:08<8:03:12, 10.37s/it]
  6%|▋         | 190/2984 [34:18<8:03:20, 10.38s/it]
                                
0: {'loss': 0.5204, 'grad_norm': 1.3912175267983369, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.04}
0: {'loss': 0.4934, 'grad_norm': 0.9257542870608921, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.04}
0:                     

  6%|▋         | 190/2984 [34:18<8:03:20, 10.38s/it]
  6%|▋         | 191/2984 [34:29<8:03:39, 10.39s/it]
  6%|▋         | 192/2984 [34:39<8:04:08, 10.40s/it]
  6%|▋         | 193/2984 [34:50<8:03:20, 10.39s/it]
  7%|▋         | 194/2984 [35:00<8:03:01, 10.39s/it]
  7%|▋         | 195/2984 [35:10<8:02:03, 10.37s/it]
  7%|▋         | 196/2984 [35:21<8:00:29, 10.34s/it]
  7%|▋         | 197/2984 [35:31<7:59:04, 10.31s/it]
  7%|▋         | 198/2984 [35:41<7:58:26, 10.30s/it]
  7%|▋         | 199/2984 [35:52<7:59:08, 10.32s/it]
  7%|▋         | 200/2984 [36:02<7:58:33, 10.31s/it]
                                                    

  7%|▋         | 200/2984 [36:02<7:58:33, 10.31s/it]
  7%|▋         | 201/2984 [36:12<7:58:06, 10.31s/it]
  7%|▋         | 202/2984 [36:22<7:57:32, 10.30s/it]
  7%|▋         | 203/2984 [36:33<7:57:00, 10.29s/it]
  7%|▋         | 204/2984 [36:43<7:56:51, 10.29s/it]
  7%|▋         | 205/2984 [36:53<7:55:09, 10.26s/it]
  7%|▋     
0: {'loss': 0.5027, 'grad_norm': 0.7882113611923741, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.04}
0: {'loss': 0.5032, 'grad_norm': 0.7904547543901974, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.04}
0:     | 206/2984 [37:03<7:55:43, 10.27s/it]
  7%|▋         | 207/2984 [37:14<7:56:36, 10.30s/it]
  7%|▋         | 208/2984 [37:24<7:57:09, 10.31s/it]
  7%|▋         | 209/2984 [37:35<8:00:33, 10.39s/it]
  7%|▋         | 210/2984 [37:45<8:00:51, 10.40s/it]
                                                    

  7%|▋         | 210/2984 [37:45<8:00:51, 10.40s/it]
  7%|▋         | 211/2984 [37:56<8:01:10, 10.41s/it]
  7%|▋         | 212/2984 [38:06<8:03:21, 10.46s/it]
  7%|▋         | 213/2984 [38:17<8:02:18, 10.44s/it]
  7%|▋         | 214/2984 [38:27<8:03:01, 10.46s/it]
  7%|▋         | 215/2984 [38:38<8:03:22, 10.47s/it]
  7%|▋         | 216/2984 [38:48<8:00:34, 10.42s/it]
  7%|▋         | 217/2984 [38:58<7:59:07, 10.39s/it]
  7%|▋         | 218/2984 [39:09<7:58:01, 10.37s/it]
  7%|▋         | 219/2984 [39:19<7:58:06, 10.37s/it]
  7%|▋         | 220/2984 [39:29<7:58:39, 10.39s/it]
                                                    

  7%|▋         | 220/2984 [39:29<7:58:39, 10.39
0: {'loss': 0.4914, 'grad_norm': 0.7699762451747684, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.05}
0: s/it]
  7%|▋         | 221/2984 [39:40<7:59:32, 10.41s/it]
  7%|▋         | 222/2984 [39:50<7:58:18, 10.39s/it]
  7%|▋         | 223/2984 [40:00<7:56:50, 10.36s/it]
  8%|▊         | 224/2984 [40:11<7:55:27, 10.34s/it]
  8%|▊         | 225/2984 [40:21<7:55:41, 10.34s/it]
  8%|▊         | 226/2984 [40:31<7:55:20, 10.34s/it]
  8%|▊         | 227/2984 [40:42<7:55:44, 10.35s/it]
  8%|▊         | 228/2984 [40:52<7:54:19, 10.33s/it]
  8%|▊         | 229/2984 [41:02<7:52:41, 10.29s/it]
  8%|▊         | 230/2984 [41:12<7:51:24, 10.27s/it]
                                                    

  8%|▊         | 230/2984 [41:13<7:51:24, 10.27s/it]
  8%|▊         | 231/2984 [41:23<7:50:41, 10.26s/it]
  8%|▊         | 232/2984 [41:33<7:50:57, 10.27s/it]
  8%|▊         | 233/2984 [41:43<7:52:15, 10.30s/it]
  8%|▊         | 234/2984 [41:54<7:53:28, 10.33s/it]
  8%|▊         | 235/2984 [42:04<7:53:53, 10.34s/it]
  8%|▊         | 236/2984 [42:15<7:53:41, 10.34s/it]
  8%|▊         | 237/2984 [
0: {'loss': 0.4949, 'grad_norm': 0.8385067668644607, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.05}
0: {'loss': 0.4952, 'grad_norm': 0.8409881251814937, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.05}
0: 42:25<7:53:48, 10.35s/it]
  8%|▊         | 238/2984 [42:35<7:55:04, 10.38s/it]
  8%|▊         | 239/2984 [42:46<7:55:54, 10.40s/it]
  8%|▊         | 240/2984 [42:56<7:56:15, 10.41s/it]
                                                    

  8%|▊         | 240/2984 [42:56<7:56:15, 10.41s/it]
  8%|▊         | 241/2984 [43:07<7:58:48, 10.47s/it]
  8%|▊         | 242/2984 [43:18<8:02:29, 10.56s/it]
  8%|▊         | 243/2984 [43:28<8:03:24, 10.58s/it]
  8%|▊         | 244/2984 [43:39<8:01:59, 10.55s/it]
  8%|▊         | 245/2984 [43:49<8:03:16, 10.59s/it]
  8%|▊         | 246/2984 [44:00<8:04:45, 10.62s/it]
  8%|▊         | 247/2984 [44:11<8:05:52, 10.65s/it]
  8%|▊         | 248/2984 [44:22<8:06:25, 10.67s/it]
  8%|▊         | 249/2984 [44:32<8:07:24, 10.69s/it]
  8%|▊         | 250/2984 [44:43<8:07:35, 10.70s/it]
                                                    

  8%|▊         | 250/2984 [44:43<8:07:35, 10.70s/it]
  8%|▊         | 251/2984 [44:54<8:05:25, 10.66s/it]
  8%|▊  
0: {'loss': 0.5021, 'grad_norm': 0.7569905826829811, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.05}
0:        | 252/2984 [45:04<8:02:47, 10.60s/it]
  8%|▊         | 253/2984 [45:14<8:00:27, 10.56s/it]
  9%|▊         | 254/2984 [45:25<7:59:06, 10.53s/it]
  9%|▊         | 255/2984 [45:35<7:58:45, 10.53s/it]
  9%|▊         | 256/2984 [45:46<7:58:15, 10.52s/it]
  9%|▊         | 257/2984 [45:57<7:59:28, 10.55s/it]
  9%|▊         | 258/2984 [46:07<8:00:42, 10.58s/it]
  9%|▊         | 259/2984 [46:18<8:01:26, 10.60s/it]
  9%|▊         | 260/2984 [46:28<8:01:37, 10.61s/it]
                                                    

  9%|▊         | 260/2984 [46:29<8:01:37, 10.61s/it]
  9%|▊         | 261/2984 [46:39<8:01:21, 10.61s/it]
  9%|▉         | 262/2984 [46:50<8:02:45, 10.64s/it]
  9%|▉         | 263/2984 [47:01<8:03:59, 10.67s/it]
  9%|▉         | 264/2984 [47:11<8:06:39, 10.73s/it]
  9%|▉         | 265/2984 [47:22<8:08:35, 10.78s/it]
  9%|▉         | 266/2984 [47:33<8:07:46, 10.77s/it]
  9%|▉         | 267/2984 [47:44<8:07:50, 10.77s/it]
  9%|▉         | 268/2984 [47:56<8:24:56, 1
0: {'loss': 0.4783, 'grad_norm': 0.7323058510962656, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.05}
0: {'loss': 0.5017, 'grad_norm': 0.8299198236529021, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.06}
0: 1.15s/it]
  9%|▉         | 269/2984 [48:07<8:17:21, 10.99s/it]
  9%|▉         | 270/2984 [48:17<8:12:19, 10.88s/it]
                                                    

  9%|▉         | 270/2984 [48:17<8:12:19, 10.88s/it]
  9%|▉         | 271/2984 [48:28<8:08:52, 10.81s/it]
  9%|▉         | 272/2984 [48:39<8:07:45, 10.79s/it]
  9%|▉         | 273/2984 [48:49<8:05:58, 10.76s/it]
  9%|▉         | 274/2984 [49:00<8:03:51, 10.71s/it]
  9%|▉         | 275/2984 [49:10<8:01:48, 10.67s/it]
  9%|▉         | 276/2984 [49:21<8:00:08, 10.64s/it]
  9%|▉         | 277/2984 [49:31<7:57:02, 10.57s/it]
  9%|▉         | 278/2984 [49:42<7:55:35, 10.55s/it]
  9%|▉         | 279/2984 [49:52<7:54:51, 10.53s/it]
  9%|▉         | 280/2984 [50:03<7:55:08, 10.54s/it]
                                                    

  9%|▉         | 280/2984 [50:03<7:55:08, 10.54s/it]
  9%|▉         | 281/2984 [50:13<7:54:45, 10.54s/it]
  9%|▉         | 282/2984 [50:24<7:54:39, 10.54s/it]
  9%|▉         | 283/298
0: {'loss': 0.4899, 'grad_norm': 0.7374023088066498, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.06}
0: 4 [50:34<7:53:48, 10.52s/it]
 10%|▉         | 284/2984 [50:45<7:50:13, 10.45s/it]
 10%|▉         | 285/2984 [50:55<7:46:51, 10.38s/it]
 10%|▉         | 286/2984 [51:05<7:44:13, 10.32s/it]
 10%|▉         | 287/2984 [51:15<7:42:29, 10.29s/it]
 10%|▉         | 288/2984 [51:26<7:42:41, 10.30s/it]
 10%|▉         | 289/2984 [51:36<7:42:30, 10.30s/it]
 10%|▉         | 290/2984 [51:46<7:42:24, 10.30s/it]
                                                    

 10%|▉         | 290/2984 [51:46<7:42:24, 10.30s/it]
 10%|▉         | 291/2984 [51:57<7:41:57, 10.29s/it]
 10%|▉         | 292/2984 [52:07<7:41:29, 10.29s/it]
 10%|▉         | 293/2984 [52:17<7:40:09, 10.26s/it]
 10%|▉         | 294/2984 [52:27<7:39:06, 10.24s/it]
 10%|▉         | 295/2984 [52:37<7:38:34, 10.23s/it]
 10%|▉         | 296/2984 [52:48<7:42:19, 10.32s/it]
 10%|▉         | 297/2984 [53:01<8:22:34, 11.22s/it]
 10%|▉         | 298/2984 [53:11<8:07:59, 10.90s/it]
 10%|█         | 299/2984 [53:22<8:00:28, 10.74s/it]
 10%|�
0: {'loss': 0.4834, 'grad_norm': 0.788449317387669, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.06}
0: {'loss': 0.6693, 'grad_norm': 1.710370199802714, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.06}
0: ��         | 300/2984 [53:32<7:54:26, 10.61s/it]
                                                    

 10%|█         | 300/2984 [53:32<7:54:26, 10.61s/it]
 10%|█         | 301/2984 [53:42<7:48:40, 10.48s/it]
 10%|█         | 302/2984 [53:52<7:43:19, 10.37s/it]
 10%|█         | 303/2984 [54:02<7:39:04, 10.27s/it]
 10%|█         | 304/2984 [54:13<7:37:06, 10.23s/it]
 10%|█         | 305/2984 [54:23<7:36:07, 10.22s/it]
 10%|█         | 306/2984 [54:33<7:35:22, 10.20s/it]
 10%|█         | 307/2984 [54:43<7:34:47, 10.19s/it]
 10%|█         | 308/2984 [54:53<7:33:20, 10.16s/it]
 10%|█         | 309/2984 [55:03<7:33:16, 10.17s/it]
 10%|█         | 310/2984 [55:14<7:32:55, 10.16s/it]
                                                    

 10%|█         | 310/2984 [55:14<7:32:55, 10.16s/it]
 10%|█         | 311/2984 [55:24<7:33:07, 10.17s/it]
 10%|█         | 312/2984 [55:34<7:33:14, 10.18s/it]
 10%|█         | 313/2984 [55:44<7:33:21, 10.18s/it]
 11%|█         | 314/2984 [55:54<7:32:18
0: {'loss': 0.7518, 'grad_norm': 2.5392125522086095, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.06}
0: , 10.16s/it]
 11%|█         | 315/2984 [56:04<7:32:07, 10.16s/it]
 11%|█         | 316/2984 [56:15<7:32:53, 10.19s/it]
 11%|█         | 317/2984 [56:25<7:33:29, 10.20s/it]
 11%|█         | 318/2984 [56:35<7:33:15, 10.20s/it]
 11%|█         | 319/2984 [56:45<7:32:53, 10.20s/it]
 11%|█         | 320/2984 [56:55<7:31:52, 10.18s/it]
                                                    

 11%|█         | 320/2984 [56:55<7:31:52, 10.18s/it]
 11%|█         | 321/2984 [57:06<7:31:47, 10.18s/it]
 11%|█         | 322/2984 [57:16<7:31:55, 10.19s/it]
 11%|█         | 323/2984 [57:26<7:31:39, 10.18s/it]
 11%|█         | 324/2984 [57:36<7:31:51, 10.19s/it]
 11%|█         | 325/2984 [57:46<7:32:41, 10.22s/it]
 11%|█         | 326/2984 [57:57<7:32:42, 10.22s/it]
 11%|█         | 327/2984 [58:07<7:31:58, 10.21s/it]
 11%|█         | 328/2984 [58:17<7:33:28, 10.24s/it]
 11%|█         | 329/2984 [58:28<7:34:32, 10.27s/it]
 11%|█         | 330/2984 [58:38<7:36:57, 10.33s/it]
                      
0: {'loss': 0.6961, 'grad_norm': 1.3626162539020525, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.07}
0: {'loss': 0.6672, 'grad_norm': 1.2100099954181724, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.07}
0:                               

 11%|█         | 330/2984 [58:38<7:36:57, 10.33s/it]
 11%|█         | 331/2984 [58:49<7:42:32, 10.46s/it]
 11%|█         | 332/2984 [58:59<7:46:16, 10.55s/it]
 11%|█         | 333/2984 [59:10<7:49:45, 10.63s/it]
 11%|█         | 334/2984 [59:21<7:54:47, 10.75s/it]
 11%|█         | 335/2984 [59:32<7:54:30, 10.75s/it]
 11%|█▏        | 336/2984 [59:43<7:53:31, 10.73s/it]
 11%|█▏        | 337/2984 [59:54<7:55:15, 10.77s/it]
 11%|█▏        | 338/2984 [1:00:04<7:49:11, 10.64s/it]
 11%|█▏        | 339/2984 [1:00:14<7:45:06, 10.55s/it]
 11%|█▏        | 340/2984 [1:00:25<7:41:47, 10.48s/it]
                                                      

 11%|█▏        | 340/2984 [1:00:25<7:41:47, 10.48s/it]
 11%|█▏        | 341/2984 [1:00:35<7:41:44, 10.48s/it]
 11%|█▏        | 342/2984 [1:00:46<7:42:33, 10.50s/it]
 11%|█▏        | 343/2984 [1:00:56<7:42:28, 10.51s/it]
 12%|█▏        | 344/2984 [1:01:07<7:40:50, 10.47s/it]
 12%|█▏        |
0: {'loss': 0.65, 'grad_norm': 1.0882562546489025, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.07}
0:  345/2984 [1:01:17<7:40:07, 10.46s/it]
 12%|█▏        | 346/2984 [1:01:27<7:38:47, 10.43s/it]
 12%|█▏        | 347/2984 [1:01:38<7:38:23, 10.43s/it]
 12%|█▏        | 348/2984 [1:01:48<7:37:55, 10.42s/it]
 12%|█▏        | 349/2984 [1:01:59<7:38:40, 10.44s/it]
 12%|█▏        | 350/2984 [1:02:09<7:39:05, 10.46s/it]
                                                      

 12%|█▏        | 350/2984 [1:02:09<7:39:05, 10.46s/it]
 12%|█▏        | 351/2984 [1:02:20<7:37:37, 10.43s/it]
 12%|█▏        | 352/2984 [1:02:30<7:36:11, 10.40s/it]
 12%|█▏        | 353/2984 [1:02:40<7:34:40, 10.37s/it]
 12%|█▏        | 354/2984 [1:02:51<7:34:25, 10.37s/it]
 12%|█▏        | 355/2984 [1:03:01<7:34:10, 10.37s/it]
 12%|█▏        | 356/2984 [1:03:11<7:34:43, 10.38s/it]
 12%|█▏        | 357/2984 [1:03:22<7:35:43, 10.41s/it]
 12%|█▏        | 358/2984 [1:03:32<7:37:44, 10.46s/it]
 12%|█▏        | 359/2984 [1:03:43<7:37:58, 10.47s/it]
 12%|█▏        | 360/2984 [1:03:53<7:37:
0: {'loss': 0.6351, 'grad_norm': 1.1470521134114764, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.07}
0: {'loss': 0.6369, 'grad_norm': 1.927084932467842, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.07}
0: 53, 10.47s/it]
                                                      

 12%|█▏        | 360/2984 [1:03:53<7:37:53, 10.47s/it]
 12%|█▏        | 361/2984 [1:04:04<7:37:44, 10.47s/it]
 12%|█▏        | 362/2984 [1:04:14<7:37:39, 10.47s/it]
 12%|█▏        | 363/2984 [1:04:25<7:37:19, 10.47s/it]
 12%|█▏        | 364/2984 [1:04:35<7:36:21, 10.45s/it]
 12%|█▏        | 365/2984 [1:04:46<7:35:59, 10.45s/it]
 12%|█▏        | 366/2984 [1:04:56<7:35:49, 10.45s/it]
 12%|█▏        | 367/2984 [1:05:07<7:37:31, 10.49s/it]
 12%|█▏        | 368/2984 [1:05:17<7:38:19, 10.51s/it]
 12%|█▏        | 369/2984 [1:05:29<7:49:17, 10.77s/it]
 12%|█▏        | 370/2984 [1:05:39<7:47:40, 10.73s/it]
                                                      

 12%|█▏        | 370/2984 [1:05:39<7:47:40, 10.73s/it]
 12%|█▏        | 371/2984 [1:05:50<7:45:17, 10.68s/it]
 12%|█▏        | 372/2984 [1:06:00<7:43:59, 10.66s/it]
 12%|█▎        | 373/2984 [1:06:11<7:41:31, 10.61s/it]
 13%|█▎ 
0: {'loss': 0.6314, 'grad_norm': 1.6021427413243425, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.08}
0:        | 374/2984 [1:06:22<7:42:15, 10.63s/it]
 13%|█▎        | 375/2984 [1:06:32<7:39:05, 10.56s/it]
 13%|█▎        | 376/2984 [1:06:42<7:35:48, 10.49s/it]
 13%|█▎        | 377/2984 [1:06:53<7:33:19, 10.43s/it]
 13%|█▎        | 378/2984 [1:07:03<7:33:43, 10.45s/it]
 13%|█▎        | 379/2984 [1:07:14<7:34:09, 10.46s/it]
 13%|█▎        | 380/2984 [1:07:24<7:33:51, 10.46s/it]
                                                      

 13%|█▎        | 380/2984 [1:07:24<7:33:51, 10.46s/it]
 13%|█▎        | 381/2984 [1:07:35<7:34:26, 10.47s/it]
 13%|█▎        | 382/2984 [1:07:45<7:33:54, 10.47s/it]
 13%|█▎        | 383/2984 [1:07:55<7:33:24, 10.46s/it]
 13%|█▎        | 384/2984 [1:08:06<7:32:40, 10.45s/it]
 13%|█▎        | 385/2984 [1:08:16<7:29:30, 10.38s/it]
 13%|█▎        | 386/2984 [1:08:26<7:26:22, 10.31s/it]
 13%|█▎        | 387/2984 [1:08:36<7:24:26, 10.27s/it]
 13%|█▎        | 388/2984 [1:08:47<7:23:21, 10.25s/it]
 13%|█▎        | 389/2984 [1:08:
0: {'loss': 0.6216, 'grad_norm': 1.1042419011037687, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.08}
0: {'loss': 0.6165, 'grad_norm': 0.9614853339986089, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.08}
0: 57<7:23:03, 10.24s/it]
 13%|█▎        | 390/2984 [1:09:07<7:24:26, 10.28s/it]
                                                      

 13%|█▎        | 390/2984 [1:09:07<7:24:26, 10.28s/it]
 13%|█▎        | 391/2984 [1:09:17<7:24:47, 10.29s/it]
 13%|█▎        | 392/2984 [1:09:28<7:25:23, 10.31s/it]
 13%|█▎        | 393/2984 [1:09:38<7:26:29, 10.34s/it]
 13%|█▎        | 394/2984 [1:09:49<7:27:35, 10.37s/it]
 13%|█▎        | 395/2984 [1:09:59<7:27:48, 10.38s/it]
 13%|█▎        | 396/2984 [1:10:10<7:28:31, 10.40s/it]
 13%|█▎        | 397/2984 [1:10:20<7:29:51, 10.43s/it]
 13%|█▎        | 398/2984 [1:10:30<7:29:54, 10.44s/it]
 13%|█▎        | 399/2984 [1:10:41<7:29:35, 10.44s/it]
 13%|█▎        | 400/2984 [1:10:51<7:29:06, 10.43s/it]
                                                      

 13%|█▎        | 400/2984 [1:10:51<7:29:06, 10.43s/it]
 13%|█▎        | 401/2984 [1:11:02<7:27:57, 10.41s/it]
 13%|█▎        | 402/2984 [1:11:13<7:34:22, 10.56s/it]
 14%
0: {'loss': 0.6178, 'grad_norm': 1.371790981800812, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.08}
0: |█▎        | 403/2984 [1:11:23<7:30:27, 10.47s/it]
 14%|█▎        | 404/2984 [1:11:33<7:29:02, 10.44s/it]
 14%|█▎        | 405/2984 [1:11:44<7:29:56, 10.47s/it]
 14%|█▎        | 406/2984 [1:11:54<7:30:01, 10.47s/it]
 14%|█▎        | 407/2984 [1:12:05<7:30:38, 10.49s/it]
 14%|█▎        | 408/2984 [1:12:15<7:31:55, 10.53s/it]
 14%|█▎        | 409/2984 [1:12:26<7:31:50, 10.53s/it]
 14%|█▎        | 410/2984 [1:12:37<7:32:54, 10.56s/it]
                                                      

 14%|█▎        | 410/2984 [1:12:37<7:32:54, 10.56s/it]
 14%|█▍        | 411/2984 [1:12:47<7:34:17, 10.59s/it]
 14%|█▍        | 412/2984 [1:12:58<7:34:19, 10.60s/it]
 14%|█▍        | 413/2984 [1:13:08<7:31:38, 10.54s/it]
 14%|█▍        | 414/2984 [1:13:19<7:30:05, 10.51s/it]
 14%|█▍        | 415/2984 [1:13:29<7:28:54, 10.48s/it]
 14%|█▍        | 416/2984 [1:13:39<7:27:31, 10.46s/it]
 14%|█▍        | 417/2984 [1:13:50<7:24:25, 10.39s/it]
 14%|█▍        | 418/298
0: {'loss': 0.6197, 'grad_norm': 1.0764562045280033, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.08}
0: {'loss': 0.5992, 'grad_norm': 1.1296765157086175, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.09}
0: 4 [1:14:00<7:24:30, 10.39s/it]
 14%|█▍        | 419/2984 [1:14:11<7:24:47, 10.40s/it]
 14%|█▍        | 420/2984 [1:14:21<7:29:29, 10.52s/it]
                                                      

 14%|█▍        | 420/2984 [1:14:21<7:29:29, 10.52s/it]
 14%|█▍        | 421/2984 [1:14:32<7:29:08, 10.51s/it]
 14%|█▍        | 422/2984 [1:14:42<7:25:54, 10.44s/it]
 14%|█▍        | 423/2984 [1:14:52<7:21:48, 10.35s/it]
 14%|█▍        | 424/2984 [1:15:03<7:23:33, 10.40s/it]
 14%|█▍        | 425/2984 [1:15:13<7:23:20, 10.39s/it]
 14%|█▍        | 426/2984 [1:15:24<7:24:08, 10.42s/it]
 14%|█▍        | 427/2984 [1:15:34<7:24:57, 10.44s/it]
 14%|█▍        | 428/2984 [1:15:45<7:26:31, 10.48s/it]
 14%|█▍        | 429/2984 [1:15:55<7:28:41, 10.54s/it]
 14%|█▍        | 430/2984 [1:16:06<7:29:46, 10.57s/it]
                                                      

 14%|█▍        | 430/2984 [1:16:06<7:29:46, 10.57s/it]
 14%|█▍        | 431/2984 [1:16:17<7:30:51, 10.60s/
0: {'loss': 0.5967, 'grad_norm': 1.1192045491803229, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.09}
0: it]
 14%|█▍        | 432/2984 [1:16:28<7:35:07, 10.70s/it]
 15%|█▍        | 433/2984 [1:16:39<7:38:15, 10.78s/it]
 15%|█▍        | 434/2984 [1:16:50<7:40:19, 10.83s/it]
 15%|█▍        | 435/2984 [1:17:00<7:41:14, 10.86s/it]
 15%|█▍        | 436/2984 [1:17:11<7:41:22, 10.86s/it]
 15%|█▍        | 437/2984 [1:17:22<7:41:24, 10.87s/it]
 15%|█▍        | 438/2984 [1:17:33<7:41:31, 10.88s/it]
 15%|█▍        | 439/2984 [1:17:44<7:41:39, 10.88s/it]
 15%|█▍        | 440/2984 [1:17:55<7:41:20, 10.88s/it]
                                                      

 15%|█▍        | 440/2984 [1:17:55<7:41:20, 10.88s/it]
 15%|█▍        | 441/2984 [1:18:06<7:41:10, 10.88s/it]
 15%|█▍        | 442/2984 [1:18:16<7:39:19, 10.84s/it]
 15%|█▍        | 443/2984 [1:18:27<7:39:42, 10.85s/it]
 15%|█▍        | 444/2984 [1:18:38<7:39:58, 10.87s/it]
 15%|█▍        | 445/2984 [1:18:49<7:39:14, 10.85s/it]
 15%|█▍        | 446/2984 [1:19:00<7:38:37, 10.84s/it]
 15%|█▍        |
0: {'loss': 0.6017, 'grad_norm': 1.026531750523859, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.09}
0: {'loss': 0.604, 'grad_norm': 1.77749489061864, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.09}
0:  447/2984 [1:19:11<7:38:15, 10.84s/it]
 15%|█▌        | 448/2984 [1:19:22<7:38:28, 10.85s/it]
 15%|█▌        | 449/2984 [1:19:32<7:37:02, 10.82s/it]
 15%|█▌        | 450/2984 [1:19:43<7:35:57, 10.80s/it]
                                                      

 15%|█▌        | 450/2984 [1:19:43<7:35:57, 10.80s/it]
 15%|█▌        | 451/2984 [1:19:54<7:35:35, 10.79s/it]
 15%|█▌        | 452/2984 [1:20:05<7:34:05, 10.76s/it]
 15%|█▌        | 453/2984 [1:20:15<7:32:21, 10.72s/it]
 15%|█▌        | 454/2984 [1:20:26<7:31:15, 10.70s/it]
 15%|█▌        | 455/2984 [1:20:36<7:26:29, 10.59s/it]
 15%|█▌        | 456/2984 [1:20:47<7:26:08, 10.59s/it]
 15%|█▌        | 457/2984 [1:20:57<7:23:11, 10.52s/it]
 15%|█▌        | 458/2984 [1:21:07<7:20:18, 10.46s/it]
 15%|█▌        | 459/2984 [1:21:18<7:17:58, 10.41s/it]
 15%|█▌        | 460/2984 [1:21:28<7:16:33, 10.38s/it]
                                                      

 15%|█▌        | 460/2984 [1:21:28<7:16:33,
0: {'loss': 0.5893, 'grad_norm': 0.9727537246174046, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.09}
0:  10.38s/it]
 15%|█▌        | 461/2984 [1:21:38<7:15:17, 10.35s/it]
 15%|█▌        | 462/2984 [1:21:49<7:15:22, 10.36s/it]
 16%|█▌        | 463/2984 [1:21:59<7:14:56, 10.35s/it]
 16%|█▌        | 464/2984 [1:22:09<7:14:05, 10.34s/it]
 16%|█▌        | 465/2984 [1:22:20<7:14:41, 10.35s/it]
 16%|█▌        | 466/2984 [1:22:30<7:15:14, 10.37s/it]
 16%|█▌        | 467/2984 [1:22:41<7:15:39, 10.39s/it]
 16%|█▌        | 468/2984 [1:22:51<7:15:41, 10.39s/it]
 16%|█▌        | 469/2984 [1:23:01<7:15:37, 10.39s/it]
 16%|█▌        | 470/2984 [1:23:12<7:16:33, 10.42s/it]
                                                      

 16%|█▌        | 470/2984 [1:23:12<7:16:33, 10.42s/it]
 16%|█▌        | 471/2984 [1:23:22<7:17:45, 10.45s/it]
 16%|█▌        | 472/2984 [1:23:33<7:17:37, 10.45s/it]
 16%|█▌        | 473/2984 [1:23:43<7:16:46, 10.44s/it]
 16%|█▌        | 474/2984 [1:23:54<7:16:23, 10.43s/it]
 16%|█▌        | 475/2984 [1:24:04<7:13:58, 10.38s/it]
 16%|█▌ 
0: {'loss': 0.5856, 'grad_norm': 0.9008448886798003, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.1}
0:        | 476/2984 [1:24:14<7:10:57, 10.31s/it]
 16%|█▌        | 477/2984 [1:24:24<7:09:30, 10.28s/it]
 16%|█▌        | 478/2984 [1:24:35<7:09:39, 10.29s/it]
 16%|█▌        | 479/2984 [1:24:45<7:09:40, 10.29s/it]
 16%|█▌        | 480/2984 [1:24:55<7:08:36, 10.27s/it]
                                                      

 16%|█▌        | 480/2984 [1:24:55<7:08:36, 10.27s/it]
 16%|█▌        | 481/2984 [1:25:06<7:10:40, 10.32s/it]
 16%|█▌        | 482/2984 [1:25:16<7:11:28, 10.35s/it]
 16%|█▌        | 483/2984 [1:25:26<7:12:15, 10.37s/it]
 16%|█▌        | 484/2984 [1:25:37<7:12:19, 10.38s/it]
 16%|█▋        | 485/2984 [1:25:48<7:17:50, 10.51s/it]
 16%|█▋        | 486/2984 [1:25:58<7:19:13, 10.55s/it]
 16%|█▋        | 487/2984 [1:26:09<7:20:08, 10.58s/it]
 16%|█▋        | 488/2984 [1:26:20<7:23:19, 10.66s/it]
 16%|█▋        | 489/2984 [1:26:30<7:22:29, 10.64s/it]
 16%|█▋        | 490/2984 [1:26:41<7:20:51, 10.61s/it]
                                    
0: {'loss': 0.5857, 'grad_norm': 0.9842448919723004, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.1}
0: {'loss': 0.6022, 'grad_norm': 1.031112420426246, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.1}
0:                   

 16%|█▋        | 490/2984 [1:26:41<7:20:51, 10.61s/it]
 16%|█▋        | 491/2984 [1:26:51<7:20:31, 10.60s/it]
 16%|█▋        | 492/2984 [1:27:02<7:16:54, 10.52s/it]
 17%|█▋        | 493/2984 [1:27:12<7:12:41, 10.42s/it]
 17%|█▋        | 494/2984 [1:27:22<7:09:03, 10.34s/it]
 17%|█▋        | 495/2984 [1:27:32<7:07:45, 10.31s/it]
 17%|█▋        | 496/2984 [1:27:43<7:06:34, 10.29s/it]
 17%|█▋        | 497/2984 [1:27:53<7:06:24, 10.29s/it]
 17%|█▋        | 498/2984 [1:28:03<7:05:28, 10.27s/it]
 17%|█▋        | 499/2984 [1:28:13<7:06:52, 10.31s/it]
 17%|█▋        | 500/2984 [1:28:24<7:07:50, 10.33s/it]
                                                      

 17%|█▋        | 500/2984 [1:28:24<7:07:50, 10.33s/it]
 17%|█▋        | 501/2984 [1:28:34<7:07:06, 10.32s/it]
 17%|█▋        | 502/2984 [1:28:45<7:07:03, 10.32s/it]
 17%|█▋        | 503/2984 [1:28:55<7:05:43, 10.30s/it]
 17%|█▋        | 504/2984 [1:29:05<7:05:53, 10.30s/it]
 17%
0: {'loss': 0.5862, 'grad_norm': 0.9860314142985375, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.1}
0: |█▋        | 505/2984 [1:29:16<7:14:05, 10.51s/it]
 17%|█▋        | 506/2984 [1:29:26<7:12:38, 10.48s/it]
 17%|█▋        | 507/2984 [1:29:37<7:11:35, 10.45s/it]
 17%|█▋        | 508/2984 [1:29:47<7:10:59, 10.44s/it]
 17%|█▋        | 509/2984 [1:29:58<7:09:32, 10.41s/it]
 17%|█▋        | 510/2984 [1:30:08<7:07:27, 10.37s/it]
                                                      

 17%|█▋        | 510/2984 [1:30:08<7:07:27, 10.37s/it]
 17%|█▋        | 511/2984 [1:30:18<7:05:54, 10.33s/it]
 17%|█▋        | 512/2984 [1:30:28<7:03:06, 10.27s/it]
 17%|█▋        | 513/2984 [1:30:39<7:02:45, 10.27s/it]
 17%|█▋        | 514/2984 [1:30:49<7:03:08, 10.28s/it]
 17%|█▋        | 515/2984 [1:30:59<7:03:04, 10.28s/it]
 17%|█▋        | 516/2984 [1:31:09<7:02:33, 10.27s/it]
 17%|█▋        | 517/2984 [1:31:20<7:02:42, 10.28s/it]
 17%|█▋        | 518/2984 [1:31:30<7:03:36, 10.31s/it]
 17%|█▋        | 519/2984 [1:31:40<7:01:26, 10.26s/it]
 17%|█▋        | 520/298
0: {'loss': 0.5864, 'grad_norm': 1.103949252805899, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.1}
0: {'loss': 0.5798, 'grad_norm': 1.0686935565481341, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.11}
0: 4 [1:31:50<7:00:38, 10.24s/it]
                                                      

 17%|█▋        | 520/2984 [1:31:50<7:00:38, 10.24s/it]
 17%|█▋        | 521/2984 [1:32:01<7:00:17, 10.24s/it]
 17%|█▋        | 522/2984 [1:32:11<7:01:07, 10.26s/it]
 18%|█▊        | 523/2984 [1:32:21<7:01:43, 10.28s/it]
 18%|█▊        | 524/2984 [1:32:32<7:02:07, 10.30s/it]
 18%|█▊        | 525/2984 [1:32:42<7:02:23, 10.31s/it]
 18%|█▊        | 526/2984 [1:32:52<7:03:59, 10.35s/it]
 18%|█▊        | 527/2984 [1:33:03<7:06:12, 10.41s/it]
 18%|█▊        | 528/2984 [1:33:13<7:06:13, 10.41s/it]
 18%|█▊        | 529/2984 [1:33:24<7:05:29, 10.40s/it]
 18%|█▊        | 530/2984 [1:33:34<7:05:11, 10.40s/it]
                                                      

 18%|█▊        | 530/2984 [1:33:34<7:05:11, 10.40s/it]
 18%|█▊        | 531/2984 [1:33:44<7:01:29, 10.31s/it]
 18%|█▊        | 532/2984 [1:33:54<6:58:20, 10.24s/it]
 18%|█▊        | 533/2984 [1:34:04<6:56:19, 10.19s/
0: {'loss': 0.5937, 'grad_norm': 5.406267805911325, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.11}
0: it]
 18%|█▊        | 534/2984 [1:34:14<6:54:09, 10.14s/it]
 18%|█▊        | 535/2984 [1:34:24<6:51:57, 10.09s/it]
 18%|█▊        | 536/2984 [1:34:34<6:52:07, 10.10s/it]
 18%|█▊        | 537/2984 [1:34:45<6:52:09, 10.11s/it]
 18%|█▊        | 538/2984 [1:34:55<6:49:50, 10.05s/it]
 18%|█▊        | 539/2984 [1:35:04<6:48:06, 10.01s/it]
 18%|█▊        | 540/2984 [1:35:14<6:47:06,  9.99s/it]
                                                      

 18%|█▊        | 540/2984 [1:35:14<6:47:06,  9.99s/it]
 18%|█▊        | 541/2984 [1:35:24<6:47:12, 10.00s/it]
 18%|█▊        | 542/2984 [1:35:34<6:46:04,  9.98s/it]
 18%|█▊        | 543/2984 [1:35:44<6:45:41,  9.97s/it]
 18%|█▊        | 544/2984 [1:35:54<6:45:09,  9.96s/it]
 18%|█▊        | 545/2984 [1:36:04<6:46:37, 10.00s/it]
 18%|█▊        | 546/2984 [1:36:15<6:48:41, 10.06s/it]
 18%|█▊        | 547/2984 [1:36:25<6:49:59, 10.09s/it]
 18%|█▊        | 548/2984 [1:36:35<6:50:59, 10.12s/it]
 18%|█▊        |
0: {'loss': 0.5713, 'grad_norm': 1.0354304556137872, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.11}
0: {'loss': 0.5768, 'grad_norm': 1.0040567610390394, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.11}
0:  549/2984 [1:36:45<6:50:46, 10.12s/it]
 18%|█▊        | 550/2984 [1:36:55<6:49:56, 10.11s/it]
                                                      

 18%|█▊        | 550/2984 [1:36:55<6:49:56, 10.11s/it]
 18%|█▊        | 551/2984 [1:37:05<6:49:28, 10.10s/it]
 18%|█▊        | 552/2984 [1:37:16<6:52:25, 10.18s/it]
 19%|█▊        | 553/2984 [1:37:25<6:49:02, 10.10s/it]
 19%|█▊        | 554/2984 [1:37:35<6:46:52, 10.05s/it]
 19%|█▊        | 555/2984 [1:37:45<6:45:31, 10.02s/it]
 19%|█▊        | 556/2984 [1:37:55<6:43:55,  9.98s/it]
 19%|█▊        | 557/2984 [1:38:05<6:45:14, 10.02s/it]
 19%|█▊        | 558/2984 [1:38:15<6:46:42, 10.06s/it]
 19%|█▊        | 559/2984 [1:38:25<6:45:47, 10.04s/it]
 19%|█▉        | 560/2984 [1:38:35<6:44:37, 10.02s/it]
                                                      

 19%|█▉        | 560/2984 [1:38:35<6:44:37, 10.02s/it]
 19%|█▉        | 561/2984 [1:38:45<6:43:36,  9.99s/it]
 19%|█▉        | 562/2984 [1:38:55<6:42:33,
0: {'loss': 0.5898, 'grad_norm': 1.016337114791256, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.11}
0:   9.97s/it]
 19%|█▉        | 563/2984 [1:39:05<6:42:11,  9.97s/it]
 19%|█▉        | 564/2984 [1:39:15<6:41:20,  9.95s/it]
 19%|█▉        | 565/2984 [1:39:25<6:41:03,  9.95s/it]
 19%|█▉        | 566/2984 [1:39:35<6:40:30,  9.94s/it]
 19%|█▉        | 567/2984 [1:39:45<6:40:07,  9.93s/it]
 19%|█▉        | 568/2984 [1:39:55<6:39:30,  9.92s/it]
 19%|█▉        | 569/2984 [1:40:05<6:39:13,  9.92s/it]
 19%|█▉        | 570/2984 [1:40:15<6:39:14,  9.92s/it]
                                                      

 19%|█▉        | 570/2984 [1:40:15<6:39:14,  9.92s/it]
 19%|█▉        | 571/2984 [1:40:25<6:40:08,  9.95s/it]
 19%|█▉        | 572/2984 [1:40:35<6:40:22,  9.96s/it]
 19%|█▉        | 573/2984 [1:40:45<6:40:53,  9.98s/it]
 19%|█▉        | 574/2984 [1:40:55<6:40:24,  9.97s/it]
 19%|█▉        | 575/2984 [1:41:05<6:39:55,  9.96s/it]
 19%|█▉        | 576/2984 [1:41:15<6:40:05,  9.97s/it]
 19%|█▉        | 577/2984 [1:41:25<6:40:54,  9.99s/it]
 19%|█▉ 
0: {'loss': 0.5864, 'grad_norm': 1.0138363193228443, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.12}
0: {'loss': 0.5794, 'grad_norm': 0.9377397807377607, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.12}
0:        | 578/2984 [1:41:35<6:41:35, 10.01s/it]
 19%|█▉        | 579/2984 [1:41:45<6:42:53, 10.05s/it]
 19%|█▉        | 580/2984 [1:41:55<6:43:53, 10.08s/it]
                                                      

 19%|█▉        | 580/2984 [1:41:55<6:43:53, 10.08s/it]
 19%|█▉        | 581/2984 [1:42:05<6:43:55, 10.09s/it]
 20%|█▉        | 582/2984 [1:42:15<6:42:56, 10.06s/it]
 20%|█▉        | 583/2984 [1:42:25<6:41:43, 10.04s/it]
 20%|█▉        | 584/2984 [1:42:35<6:40:30, 10.01s/it]
 20%|█▉        | 585/2984 [1:42:45<6:39:29,  9.99s/it]
 20%|█▉        | 586/2984 [1:42:55<6:39:28, 10.00s/it]
 20%|█▉        | 587/2984 [1:43:05<6:39:55, 10.01s/it]
 20%|█▉        | 588/2984 [1:43:15<6:40:08, 10.02s/it]
 20%|█▉        | 589/2984 [1:43:25<6:39:57, 10.02s/it]
 20%|█▉        | 590/2984 [1:43:35<6:39:05, 10.00s/it]
                                                      

 20%|█▉        | 590/2984 [1:43:35<6:39:05, 10.00s/it]
 20%|█▉        | 591/2984 [1:43:45<
0: {'loss': 0.5709, 'grad_norm': 1.1735379365435954, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.12}
0: 6:38:45, 10.00s/it]
 20%|█▉        | 592/2984 [1:43:55<6:38:02,  9.98s/it]
 20%|█▉        | 593/2984 [1:44:05<6:37:38,  9.98s/it]
 20%|█▉        | 594/2984 [1:44:15<6:37:15,  9.97s/it]
 20%|█▉        | 595/2984 [1:44:25<6:37:36,  9.99s/it]
 20%|█▉        | 596/2984 [1:44:35<6:37:25,  9.99s/it]
 20%|██        | 597/2984 [1:44:45<6:38:00, 10.00s/it]
 20%|██        | 598/2984 [1:44:55<6:40:36, 10.07s/it]
 20%|██        | 599/2984 [1:45:05<6:38:44, 10.03s/it]
 20%|██        | 600/2984 [1:45:15<6:37:29, 10.00s/it]
                                                      

 20%|██        | 600/2984 [1:45:15<6:37:29, 10.00s/it]
 20%|██        | 601/2984 [1:45:25<6:37:20, 10.00s/it]
 20%|██        | 602/2984 [1:45:35<6:37:39, 10.02s/it]
 20%|██        | 603/2984 [1:45:45<6:38:26, 10.04s/it]
 20%|██        | 604/2984 [1:45:55<6:39:22, 10.07s/it]
 20%|██        | 605/2984 [1:46:05<6:39:56, 10.09s/it]
 20%|██        | 606/2984 [1:46:16<6:41:05, 10.12s/it]
 20%
0: {'loss': 0.5684, 'grad_norm': 1.0218732732320062, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.12}
0: {'loss': 0.5658, 'grad_norm': 0.9425931277825617, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.12}
0: |██        | 607/2984 [1:46:26<6:41:09, 10.13s/it]
 20%|██        | 608/2984 [1:46:36<6:42:05, 10.15s/it]
 20%|██        | 609/2984 [1:46:46<6:42:52, 10.18s/it]
 20%|██        | 610/2984 [1:46:56<6:42:58, 10.18s/it]
                                                      

 20%|██        | 610/2984 [1:46:56<6:42:58, 10.18s/it]
 20%|██        | 611/2984 [1:47:07<6:42:53, 10.19s/it]
 21%|██        | 612/2984 [1:47:17<6:41:52, 10.17s/it]
 21%|██        | 613/2984 [1:47:27<6:41:54, 10.17s/it]
 21%|██        | 614/2984 [1:47:37<6:41:57, 10.18s/it]
 21%|██        | 615/2984 [1:47:47<6:41:50, 10.18s/it]
 21%|██        | 616/2984 [1:47:57<6:41:25, 10.17s/it]
 21%|██        | 617/2984 [1:48:08<6:40:40, 10.16s/it]
 21%|██        | 618/2984 [1:48:18<6:40:53, 10.17s/it]
 21%|██        | 619/2984 [1:48:28<6:40:13, 10.15s/it]
 21%|██        | 620/2984 [1:48:38<6:39:35, 10.14s/it]
                                                      

 21%|██        | 620/2984 [
0: {'loss': 0.5714, 'grad_norm': 1.1778653380766264, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.13}
0: 1:48:38<6:39:35, 10.14s/it]
 21%|██        | 621/2984 [1:48:48<6:39:09, 10.14s/it]
 21%|██        | 622/2984 [1:48:58<6:38:16, 10.12s/it]
 21%|██        | 623/2984 [1:49:08<6:38:14, 10.12s/it]
 21%|██        | 624/2984 [1:49:19<6:38:54, 10.14s/it]
 21%|██        | 625/2984 [1:49:29<6:38:11, 10.13s/it]
 21%|██        | 626/2984 [1:49:39<6:37:38, 10.12s/it]
 21%|██        | 627/2984 [1:49:49<6:37:18, 10.11s/it]
 21%|██        | 628/2984 [1:49:59<6:37:37, 10.13s/it]
 21%|██        | 629/2984 [1:50:09<6:38:39, 10.16s/it]
 21%|██        | 630/2984 [1:50:20<6:42:07, 10.25s/it]
                                                      

 21%|██        | 630/2984 [1:50:20<6:42:07, 10.25s/it]
 21%|██        | 631/2984 [1:50:30<6:42:06, 10.25s/it]
 21%|██        | 632/2984 [1:50:40<6:42:06, 10.26s/it]
 21%|██        | 633/2984 [1:50:50<6:41:42, 10.25s/it]
 21%|██        | 634/2984 [1:51:01<6:41:40, 10.26s/it]
 21%|██▏       | 635/2984 [1:51:11<6:41:53, 10.27
0: {'loss': 0.5709, 'grad_norm': 1.1064339520345101, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.13}
0: s/it]
 21%|██▏       | 636/2984 [1:51:21<6:41:46, 10.27s/it]
 21%|██▏       | 637/2984 [1:51:31<6:41:20, 10.26s/it]
 21%|██▏       | 638/2984 [1:51:42<6:39:57, 10.23s/it]
 21%|██▏       | 639/2984 [1:51:52<6:39:20, 10.22s/it]
 21%|██▏       | 640/2984 [1:52:02<6:39:21, 10.22s/it]
                                                      

 21%|██▏       | 640/2984 [1:52:02<6:39:21, 10.22s/it]
 21%|██▏       | 641/2984 [1:52:12<6:38:38, 10.21s/it]
 22%|██▏       | 642/2984 [1:52:22<6:37:42, 10.19s/it]
 22%|██▏       | 643/2984 [1:52:32<6:36:21, 10.16s/it]
 22%|██▏       | 644/2984 [1:52:43<6:36:00, 10.15s/it]
 22%|██▏       | 645/2984 [1:52:53<6:37:20, 10.19s/it]
 22%|██▏       | 646/2984 [1:53:03<6:36:26, 10.17s/it]
 22%|██▏       | 647/2984 [1:53:13<6:38:44, 10.24s/it]
 22%|██▏       | 648/2984 [1:53:24<6:38:24, 10.23s/it]
 22%|██▏       | 649/2984 [1:53:34<6:39:27, 10.26s/it]
 22%|██▏       | 650/2984 [1:53:44<6:36:5
0: {'loss': 0.5635, 'grad_norm': 0.9818438603313867, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.13}
0: {'loss': 0.5768, 'grad_norm': 1.0584022482849584, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.13}
0: 4, 10.20s/it]
                                                      

 22%|██▏       | 650/2984 [1:53:44<6:36:54, 10.20s/it]
 22%|██▏       | 651/2984 [1:53:54<6:34:39, 10.15s/it]
 22%|██▏       | 652/2984 [1:54:04<6:32:31, 10.10s/it]
 22%|██▏       | 653/2984 [1:54:14<6:31:05, 10.07s/it]
 22%|██▏       | 654/2984 [1:54:24<6:30:06, 10.05s/it]
 22%|██▏       | 655/2984 [1:54:34<6:30:04, 10.05s/it]
 22%|██▏       | 656/2984 [1:54:44<6:31:02, 10.08s/it]
 22%|██▏       | 657/2984 [1:54:54<6:30:43, 10.07s/it]
 22%|██▏       | 658/2984 [1:55:04<6:30:21, 10.07s/it]
 22%|██▏       | 659/2984 [1:55:14<6:29:10, 10.04s/it]
 22%|██▏       | 660/2984 [1:55:24<6:29:04, 10.05s/it]
                                                      

 22%|██▏       | 660/2984 [1:55:24<6:29:04, 10.05s/it]
 22%|██▏       | 661/2984 [1:55:34<6:28:30, 10.03s/it]
 22%|██▏       | 662/2984 [1:55:44<6:28:52, 10.05s/it]
 22%|██▏       | 663/2984 [1:55:54<6:2
0: {'loss': 0.5818, 'grad_norm': 1.1309691429330408, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.13}
0: 8:08, 10.03s/it]
 22%|██▏       | 664/2984 [1:56:05<6:27:48, 10.03s/it]
 22%|██▏       | 665/2984 [1:56:15<6:28:35, 10.05s/it]
 22%|██▏       | 666/2984 [1:56:25<6:29:14, 10.08s/it]
 22%|██▏       | 667/2984 [1:56:35<6:31:12, 10.13s/it]
 22%|██▏       | 668/2984 [1:56:45<6:33:32, 10.20s/it]
 22%|██▏       | 669/2984 [1:56:56<6:36:11, 10.27s/it]
 22%|██▏       | 670/2984 [1:57:06<6:39:02, 10.35s/it]
                                                      

 22%|██▏       | 670/2984 [1:57:06<6:39:02, 10.35s/it]
 22%|██▏       | 671/2984 [1:57:17<6:40:51, 10.40s/it]
 23%|██▎       | 672/2984 [1:57:27<6:41:42, 10.42s/it]
 23%|██▎       | 673/2984 [1:57:38<6:50:13, 10.65s/it]
 23%|██▎       | 674/2984 [1:57:49<6:50:45, 10.67s/it]
 23%|██▎       | 675/2984 [1:58:00<6:47:19, 10.58s/it]
 23%|██▎       | 676/2984 [1:58:10<6:44:04, 10.50s/it]
 23%|██▎       | 677/2984 [1:58:20<6:40:53, 10.43s/it]
 23%|██▎       | 678/2984 [1:5
0: {'loss': 0.5645, 'grad_norm': 1.01307653828207, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.14}
0: {'loss': 0.5563, 'grad_norm': 0.9499745577328931, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.14}
0: 8:30<6:38:22, 10.37s/it]
 23%|██▎       | 679/2984 [1:58:41<6:36:43, 10.33s/it]
 23%|██▎       | 680/2984 [1:58:51<6:35:00, 10.29s/it]
                                                      

 23%|██▎       | 680/2984 [1:58:51<6:35:00, 10.29s/it]
 23%|██▎       | 681/2984 [1:59:01<6:33:25, 10.25s/it]
 23%|██▎       | 682/2984 [1:59:11<6:32:18, 10.23s/it]
 23%|██▎       | 683/2984 [1:59:21<6:31:51, 10.22s/it]
 23%|██▎       | 684/2984 [1:59:32<6:31:53, 10.22s/it]
 23%|██▎       | 685/2984 [1:59:42<6:31:59, 10.23s/it]
 23%|██▎       | 686/2984 [1:59:52<6:31:13, 10.21s/it]
 23%|██▎       | 687/2984 [2:00:02<6:30:21, 10.20s/it]
 23%|██▎       | 688/2984 [2:00:12<6:29:58, 10.19s/it]
 23%|██▎       | 689/2984 [2:00:23<6:29:58, 10.20s/it]
 23%|██▎       | 690/2984 [2:00:33<6:29:45, 10.19s/it]
                                                      

 23%|██▎       | 690/2984 [2:00:33<6:29:45, 10.19s/it]
 23%|██▎       | 691/2984 [
0: {'loss': 0.5627, 'grad_norm': 1.0997262472089036, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.14}
0: 2:00:43<6:29:28, 10.19s/it]
 23%|██▎       | 692/2984 [2:00:53<6:29:57, 10.21s/it]
 23%|██▎       | 693/2984 [2:01:03<6:30:20, 10.22s/it]
 23%|██▎       | 694/2984 [2:01:14<6:30:55, 10.24s/it]
 23%|██▎       | 695/2984 [2:01:24<6:30:34, 10.24s/it]
 23%|██▎       | 696/2984 [2:01:34<6:30:21, 10.24s/it]
 23%|██▎       | 697/2984 [2:01:44<6:29:53, 10.23s/it]
 23%|██▎       | 698/2984 [2:01:55<6:29:36, 10.23s/it]
 23%|██▎       | 699/2984 [2:02:05<6:29:33, 10.23s/it]
 23%|██▎       | 700/2984 [2:02:15<6:29:35, 10.23s/it]
                                                      

 23%|██▎       | 700/2984 [2:02:15<6:29:35, 10.23s/it]
 23%|██▎       | 701/2984 [2:02:25<6:29:55, 10.25s/it]
 24%|██▎       | 702/2984 [2:02:36<6:29:03, 10.23s/it]
 24%|██▎       | 703/2984 [2:02:46<6:28:52, 10.23s/it]
 24%|██▎       | 704/2984 [2:02:56<6:29:50, 10.26s/it]
 24%|██▎       | 705/2984 [2:03:06<6:28:17, 10.22s/it]
 24%|██▎       | 70
0: {'loss': 0.5639, 'grad_norm': 1.4916580397855266, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.14}
0: 6/2984 [2:03:16<6:26:45, 10.19s/it]
 24%|██▎       | 707/2984 [2:03:26<6:24:42, 10.14s/it]
 24%|██▎       | 708/2984 [2:03:36<6:23:30, 10.11s/it]
 24%|██▍       | 709/2984 [2:03:46<6:21:39, 10.07s/it]
 24%|██▍       | 710/2984 [2:03:56<6:20:29, 10.04s/it]
                                                      

 24%|██▍       | 710/2984 [2:03:56<6:20:29, 10.04s/it]
 24%|██▍       | 711/2984 [2:04:06<6:19:41, 10.02s/it]
 24%|██▍       | 712/2984 [2:04:16<6:19:57, 10.03s/it]
 24%|██▍       | 713/2984 [2:04:26<6:20:06, 10.04s/it]
 24%|██▍       | 714/2984 [2:04:37<6:19:58, 10.04s/it]
 24%|██▍       | 715/2984 [2:04:47<6:21:10, 10.08s/it]
 24%|██▍       | 716/2984 [2:04:57<6:23:34, 10.15s/it]
 24%|██▍       | 717/2984 [2:05:07<6:25:19, 10.20s/it]
 24%|██▍       | 718/2984 [2:05:18<6:25:41, 10.21s/it]
 24%|██▍       | 719/2984 [2:05:28<6:25:24, 10.21s/it]
 24%|██▍       | 720/2984 [2:05:38<6:25:12, 10.21s/it]
                 
0: {'loss': 0.5621, 'grad_norm': 1.0014257806332698, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.14}
0: {'loss': 0.5581, 'grad_norm': 3.029184225974036, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.15}
0:                                      

 24%|██▍       | 720/2984 [2:05:38<6:25:12, 10.21s/it]
 24%|██▍       | 721/2984 [2:05:48<6:24:36, 10.20s/it]
 24%|██▍       | 722/2984 [2:05:58<6:21:49, 10.13s/it]
 24%|██▍       | 723/2984 [2:06:08<6:19:17, 10.07s/it]
 24%|██▍       | 724/2984 [2:06:18<6:17:38, 10.03s/it]
 24%|██▍       | 725/2984 [2:06:28<6:17:03, 10.01s/it]
 24%|██▍       | 726/2984 [2:06:38<6:19:01, 10.07s/it]
 24%|██▍       | 727/2984 [2:06:48<6:18:27, 10.06s/it]
 24%|██▍       | 728/2984 [2:06:58<6:16:57, 10.03s/it]
 24%|██▍       | 729/2984 [2:07:08<6:16:57, 10.03s/it]
 24%|██▍       | 730/2984 [2:07:18<6:17:00, 10.04s/it]
                                                      

 24%|██▍       | 730/2984 [2:07:18<6:17:00, 10.04s/it]
 24%|██▍       | 731/2984 [2:07:28<6:17:33, 10.05s/it]
 25%|██▍       | 732/2984 [2:07:38<6:17:57, 10.07s/it]
 25%|██▍       | 733/2984 [2:07:49<6:18:44, 10.10s/it]
 25%|██▍
0: {'loss': 0.5732, 'grad_norm': 0.9978557375895384, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.15}
0:        | 734/2984 [2:07:59<6:19:26, 10.12s/it]
 25%|██▍       | 735/2984 [2:08:09<6:19:31, 10.13s/it]
 25%|██▍       | 736/2984 [2:08:19<6:19:49, 10.14s/it]
 25%|██▍       | 737/2984 [2:08:29<6:22:16, 10.21s/it]
 25%|██▍       | 738/2984 [2:08:40<6:23:46, 10.25s/it]
 25%|██▍       | 739/2984 [2:08:50<6:22:46, 10.23s/it]
 25%|██▍       | 740/2984 [2:09:00<6:23:46, 10.26s/it]
                                                      

 25%|██▍       | 740/2984 [2:09:00<6:23:46, 10.26s/it]
 25%|██▍       | 741/2984 [2:09:11<6:25:35, 10.31s/it]
 25%|██▍       | 742/2984 [2:09:21<6:25:34, 10.32s/it]
 25%|██▍       | 743/2984 [2:09:32<6:29:34, 10.43s/it]
 25%|██▍       | 744/2984 [2:09:42<6:29:35, 10.44s/it]
 25%|██▍       | 745/2984 [2:09:53<6:28:59, 10.42s/it]
 25%|██▌       | 746/2984 [2:10:03<6:27:04, 10.38s/it]
 25%|██▌       | 747/2984 [2:10:13<6:24:34, 10.32s/it]
 25%|██▌       | 748/2984 [2:10:23<6:23:14, 10.28s/it]
 25%|�
0: {'loss': 0.5673, 'grad_norm': 1.2457685572662447, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.15}
0: {'loss': 0.5553, 'grad_norm': 1.034874072974349, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.15}
0: ��█▌       | 749/2984 [2:10:33<6:20:14, 10.21s/it]
 25%|██▌       | 750/2984 [2:10:43<6:17:43, 10.14s/it]
                                                      

 25%|██▌       | 750/2984 [2:10:43<6:17:43, 10.14s/it]
 25%|██▌       | 751/2984 [2:10:53<6:15:25, 10.09s/it]
 25%|██▌       | 752/2984 [2:11:03<6:13:46, 10.05s/it]
 25%|██▌       | 753/2984 [2:11:13<6:11:51, 10.00s/it]
 25%|██▌       | 754/2984 [2:11:23<6:11:52, 10.01s/it]
 25%|██▌       | 755/2984 [2:11:33<6:11:32, 10.00s/it]
 25%|██▌       | 756/2984 [2:11:43<6:10:29,  9.98s/it]
 25%|██▌       | 757/2984 [2:11:53<6:09:36,  9.96s/it]
 25%|██▌       | 758/2984 [2:12:03<6:10:09,  9.98s/it]
 25%|██▌       | 759/2984 [2:12:13<6:09:35,  9.97s/it]
 25%|██▌       | 760/2984 [2:12:23<6:10:11,  9.99s/it]
                                                      

 25%|██▌       | 760/2984 [2:12:23<6:10:11,  9.99s/it]
 26%|██▌       | 761/2984 [2:12:33<6:10:25, 10.00s/it]
 26
0: {'loss': 0.5539, 'grad_norm': 1.0185726917754907, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.15}
0: %|██▌       | 762/2984 [2:12:43<6:10:22, 10.00s/it]
 26%|██▌       | 763/2984 [2:12:53<6:12:14, 10.06s/it]
 26%|██▌       | 764/2984 [2:13:03<6:15:27, 10.15s/it]
 26%|██▌       | 765/2984 [2:13:14<6:17:01, 10.19s/it]
 26%|██▌       | 766/2984 [2:13:24<6:18:47, 10.25s/it]
 26%|██▌       | 767/2984 [2:13:34<6:19:43, 10.28s/it]
 26%|██▌       | 768/2984 [2:13:45<6:20:24, 10.30s/it]
 26%|██▌       | 769/2984 [2:13:55<6:20:24, 10.30s/it]
 26%|██▌       | 770/2984 [2:14:05<6:20:00, 10.30s/it]
                                                      

 26%|██▌       | 770/2984 [2:14:05<6:20:00, 10.30s/it]
 26%|██▌       | 771/2984 [2:14:16<6:20:16, 10.31s/it]
 26%|██▌       | 772/2984 [2:14:26<6:20:30, 10.32s/it]
 26%|██▌       | 773/2984 [2:14:37<6:21:08, 10.34s/it]
 26%|██▌       | 774/2984 [2:14:47<6:21:54, 10.37s/it]
 26%|██▌       | 775/2984 [2:14:58<6:23:47, 10.42s/it]
 26%|██▌       | 776/2984 [2:15:08<6:25:13, 10.47s
0: {'loss': 0.5577, 'grad_norm': 1.1604944047747119, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.16}
0: /it]
 26%|██▌       | 777/2984 [2:15:18<6:24:10, 10.44s/it]
 26%|██▌       | 778/2984 [2:15:29<6:22:57, 10.42s/it]
 26%|██▌       | 779/2984 [2:15:39<6:22:14, 10.40s/it]
 26%|██▌       | 780/2984 [2:15:50<6:22:17, 10.41s/it]
                                                      

 26%|██▌       | 780/2984 [2:15:50<6:22:17, 10.41s/it]
 26%|██▌       | 781/2984 [2:16:00<6:22:53, 10.43s/it]
 26%|██▌       | 782/2984 [2:16:11<6:28:09, 10.58s/it]
 26%|██▌       | 783/2984 [2:16:22<6:31:31, 10.67s/it]
 26%|██▋       | 784/2984 [2:16:33<6:30:36, 10.65s/it]
 26%|██▋       | 785/2984 [2:16:43<6:30:05, 10.64s/it]
 26%|██▋       | 786/2984 [2:16:54<6:28:09, 10.60s/it]
 26%|██▋       | 787/2984 [2:17:04<6:24:29, 10.50s/it]
 26%|██▋       | 788/2984 [2:17:14<6:20:23, 10.39s/it]
 26%|██▋       | 789/2984 [2:17:24<6:18:35, 10.35s/it]
 26%|██▋       | 790/2984 [2:17:35<6:19:04, 10.37s/it]
                                                
0: {'loss': 0.5591, 'grad_norm': 0.9745701174620548, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.16}
0: {'loss': 0.5441, 'grad_norm': 0.9595556573436612, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.16}
0:       

 26%|██▋       | 790/2984 [2:17:35<6:19:04, 10.37s/it]
 27%|██▋       | 791/2984 [2:17:45<6:20:40, 10.42s/it]
 27%|██▋       | 792/2984 [2:17:56<6:20:45, 10.42s/it]
 27%|██▋       | 793/2984 [2:18:06<6:19:59, 10.41s/it]
 27%|██▋       | 794/2984 [2:18:17<6:22:04, 10.47s/it]
 27%|██▋       | 795/2984 [2:18:27<6:23:05, 10.50s/it]
 27%|██▋       | 796/2984 [2:18:38<6:23:08, 10.51s/it]
 27%|██▋       | 797/2984 [2:18:48<6:23:11, 10.51s/it]
 27%|██▋       | 798/2984 [2:18:59<6:20:03, 10.43s/it]
 27%|██▋       | 799/2984 [2:19:09<6:16:49, 10.35s/it]
 27%|██▋       | 800/2984 [2:19:19<6:15:31, 10.32s/it]
                                                      

 27%|██▋       | 800/2984 [2:19:19<6:15:31, 10.32s/it]
 27%|██▋       | 801/2984 [2:19:29<6:14:27, 10.29s/it]
 27%|██▋       | 802/2984 [2:19:39<6:13:22, 10.27s/it]
 27%|██▋       | 803/2984 [2:19:50<6:12:42, 10.25s/it]
 27%|██▋       | 804/2984 [2:20:00<6:15
0: {'loss': 0.5508, 'grad_norm': 0.9061884577826195, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.16}
0: :47, 10.34s/it]
 27%|██▋       | 805/2984 [2:20:11<6:17:47, 10.40s/it]
 27%|██▋       | 806/2984 [2:20:21<6:18:32, 10.43s/it]
 27%|██▋       | 807/2984 [2:20:32<6:18:36, 10.43s/it]
 27%|██▋       | 808/2984 [2:20:42<6:17:47, 10.42s/it]
 27%|██▋       | 809/2984 [2:20:52<6:17:05, 10.40s/it]
 27%|██▋       | 810/2984 [2:21:03<6:16:14, 10.38s/it]
                                                      

 27%|██▋       | 810/2984 [2:21:03<6:16:14, 10.38s/it]
 27%|██▋       | 811/2984 [2:21:13<6:15:48, 10.38s/it]
 27%|██▋       | 812/2984 [2:21:23<6:15:24, 10.37s/it]
 27%|██▋       | 813/2984 [2:21:34<6:15:00, 10.36s/it]
 27%|██▋       | 814/2984 [2:21:44<6:14:36, 10.36s/it]
 27%|██▋       | 815/2984 [2:21:54<6:14:27, 10.36s/it]
 27%|██▋       | 816/2984 [2:22:05<6:15:02, 10.38s/it]
 27%|██▋       | 817/2984 [2:22:15<6:14:30, 10.37s/it]
 27%|██▋       | 818/2984 [2:22:26<6:14:34, 10.38s/it]
 27%|██▋       | 819/2984 [2:22
0: {'loss': 0.5579, 'grad_norm': 1.412578527296233, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.16}
0: {'loss': 0.5557, 'grad_norm': 0.9316339758694673, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.17}
0: :36<6:15:55, 10.42s/it]
 27%|██▋       | 820/2984 [2:22:47<6:17:04, 10.45s/it]
                                                      

 27%|██▋       | 820/2984 [2:22:47<6:17:04, 10.45s/it]
 28%|██▊       | 821/2984 [2:22:57<6:17:44, 10.48s/it]
 28%|██▊       | 822/2984 [2:23:08<6:18:13, 10.50s/it]
 28%|██▊       | 823/2984 [2:23:18<6:16:59, 10.47s/it]
 28%|██▊       | 824/2984 [2:23:29<6:18:01, 10.50s/it]
 28%|██▊       | 825/2984 [2:23:39<6:16:38, 10.47s/it]
 28%|██▊       | 826/2984 [2:23:49<6:12:17, 10.35s/it]
 28%|██▊       | 827/2984 [2:23:59<6:08:22, 10.25s/it]
 28%|██▊       | 828/2984 [2:24:09<6:05:43, 10.18s/it]
 28%|██▊       | 829/2984 [2:24:19<6:04:05, 10.14s/it]
 28%|██▊       | 830/2984 [2:24:30<6:05:29, 10.18s/it]
                                                      

 28%|██▊       | 830/2984 [2:24:30<6:05:29, 10.18s/it]
 28%|██▊       | 831/2984 [2:24:40<6:06:59, 10.23s/it]
 28%|██▊       | 832/2984 [2
0: {'loss': 0.5532, 'grad_norm': 0.9779318749703257, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.17}
0: :24:50<6:08:45, 10.28s/it]
 28%|██▊       | 833/2984 [2:25:01<6:09:55, 10.32s/it]
 28%|██▊       | 834/2984 [2:25:11<6:10:33, 10.34s/it]
 28%|██▊       | 835/2984 [2:25:22<6:11:59, 10.39s/it]
 28%|██▊       | 836/2984 [2:25:32<6:12:23, 10.40s/it]
 28%|██▊       | 837/2984 [2:25:42<6:12:25, 10.41s/it]
 28%|██▊       | 838/2984 [2:25:53<6:11:54, 10.40s/it]
 28%|██▊       | 839/2984 [2:26:03<6:11:07, 10.38s/it]
 28%|██▊       | 840/2984 [2:26:14<6:10:48, 10.38s/it]
                                                      

 28%|██▊       | 840/2984 [2:26:14<6:10:48, 10.38s/it]
 28%|██▊       | 841/2984 [2:26:24<6:09:38, 10.35s/it]
 28%|██▊       | 842/2984 [2:26:35<6:15:06, 10.51s/it]
 28%|██▊       | 843/2984 [2:26:45<6:13:51, 10.48s/it]
 28%|██▊       | 844/2984 [2:26:57<6:29:22, 10.92s/it]
 28%|██▊       | 845/2984 [2:27:08<6:25:29, 10.81s/it]
 28%|██▊       | 846/2984 [2:27:18<6:19:00, 10.64s/it]
 28%|██▊       | 847
0: {'loss': 0.5393, 'grad_norm': 1.069273048215591, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.17}
0: {'loss': 0.5515, 'grad_norm': 0.9467671735522052, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.17}
0: /2984 [2:27:28<6:14:17, 10.51s/it]
 28%|██▊       | 848/2984 [2:27:38<6:13:34, 10.49s/it]
 28%|██▊       | 849/2984 [2:27:49<6:10:34, 10.41s/it]
 28%|██▊       | 850/2984 [2:27:59<6:08:28, 10.36s/it]
                                                      

 28%|██▊       | 850/2984 [2:27:59<6:08:28, 10.36s/it]
 29%|██▊       | 851/2984 [2:28:09<6:07:19, 10.33s/it]
 29%|██▊       | 852/2984 [2:28:19<6:05:50, 10.30s/it]
 29%|██▊       | 853/2984 [2:28:30<6:04:44, 10.27s/it]
 29%|██▊       | 854/2984 [2:28:40<6:04:14, 10.26s/it]
 29%|██▊       | 855/2984 [2:28:50<6:02:13, 10.21s/it]
 29%|██▊       | 856/2984 [2:29:00<6:01:03, 10.18s/it]
 29%|██▊       | 857/2984 [2:29:10<5:59:53, 10.15s/it]
 29%|██▉       | 858/2984 [2:29:20<5:59:08, 10.14s/it]
 29%|██▉       | 859/2984 [2:29:30<5:59:12, 10.14s/it]
 29%|██▉       | 860/2984 [2:29:41<5:58:46, 10.14s/it]
                                                      

 29%|██▉       | 
0: {'loss': 0.5595, 'grad_norm': 0.9407170395093656, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.17}
0: 860/2984 [2:29:41<5:58:46, 10.14s/it]
 29%|██▉       | 861/2984 [2:29:51<5:59:41, 10.17s/it]
 29%|██▉       | 862/2984 [2:30:01<6:00:33, 10.19s/it]
 29%|██▉       | 863/2984 [2:30:11<6:01:26, 10.22s/it]
 29%|██▉       | 864/2984 [2:30:22<6:04:49, 10.33s/it]
 29%|██▉       | 865/2984 [2:30:32<6:02:28, 10.26s/it]
 29%|██▉       | 866/2984 [2:30:42<6:00:19, 10.21s/it]
 29%|██▉       | 867/2984 [2:30:52<5:58:45, 10.17s/it]
 29%|██▉       | 868/2984 [2:31:02<5:57:19, 10.13s/it]
 29%|██▉       | 869/2984 [2:31:12<5:56:47, 10.12s/it]
 29%|██▉       | 870/2984 [2:31:22<5:56:16, 10.11s/it]
                                                      

 29%|██▉       | 870/2984 [2:31:22<5:56:16, 10.11s/it]
 29%|██▉       | 871/2984 [2:31:33<5:56:34, 10.13s/it]
 29%|██▉       | 872/2984 [2:31:43<5:56:58, 10.14s/it]
 29%|██▉       | 873/2984 [2:31:53<5:58:29, 10.19s/it]
 29%|██▉       | 874/2984 [2:32:03<5:58:45, 10.20s/it]
 29%|██▉ 
0: {'loss': 0.5543, 'grad_norm': 0.9448807586308393, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.18}
0:       | 875/2984 [2:32:14<5:59:43, 10.23s/it]
 29%|██▉       | 876/2984 [2:32:24<6:00:18, 10.26s/it]
 29%|██▉       | 877/2984 [2:32:34<6:00:57, 10.28s/it]
 29%|██▉       | 878/2984 [2:32:45<6:01:29, 10.30s/it]
 29%|██▉       | 879/2984 [2:32:55<6:01:30, 10.30s/it]
 29%|██▉       | 880/2984 [2:33:05<6:00:31, 10.28s/it]
                                                      

 29%|██▉       | 880/2984 [2:33:05<6:00:31, 10.28s/it]
 30%|██▉       | 881/2984 [2:33:15<6:00:30, 10.29s/it]
 30%|██▉       | 882/2984 [2:33:26<5:58:44, 10.24s/it]
 30%|██▉       | 883/2984 [2:33:36<5:57:54, 10.22s/it]
 30%|██▉       | 884/2984 [2:33:46<5:57:20, 10.21s/it]
 30%|██▉       | 885/2984 [2:33:56<5:57:58, 10.23s/it]
 30%|██▉       | 886/2984 [2:34:07<5:59:03, 10.27s/it]
 30%|██▉       | 887/2984 [2:34:17<5:58:47, 10.27s/it]
 30%|██▉       | 888/2984 [2:34:27<5:58:49, 10.27s/it]
 30%|██▉       | 889/2984 [2:34:37<5:58:21, 10.26s/it]
 30%|�
0: {'loss': 0.542, 'grad_norm': 1.0289732459693814, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.18}
0: {'loss': 0.5441, 'grad_norm': 1.1182110460927788, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.18}
0: �█▉       | 890/2984 [2:34:48<5:58:35, 10.28s/it]
                                                      

 30%|██▉       | 890/2984 [2:34:48<5:58:35, 10.28s/it]
 30%|██▉       | 891/2984 [2:34:58<5:58:34, 10.28s/it]
 30%|██▉       | 892/2984 [2:35:08<5:59:09, 10.30s/it]
 30%|██▉       | 893/2984 [2:35:19<5:59:41, 10.32s/it]
 30%|██▉       | 894/2984 [2:35:29<5:59:37, 10.32s/it]
 30%|██▉       | 895/2984 [2:35:39<5:59:08, 10.32s/it]
 30%|███       | 896/2984 [2:35:50<5:59:08, 10.32s/it]
 30%|███       | 897/2984 [2:36:00<5:58:45, 10.31s/it]
 30%|███       | 898/2984 [2:36:10<5:57:58, 10.30s/it]
 30%|███       | 899/2984 [2:36:20<5:57:31, 10.29s/it]
 30%|███       | 900/2984 [2:36:31<5:57:13, 10.28s/it]
                                                      

 30%|███       | 900/2984 [2:36:31<5:57:13, 10.28s/it]
 30%|███       | 901/2984 [2:36:41<5:57:55, 10.31s/it]
 30%|███       | 902/2984 [2:36:51<5:57:52, 10.31s/it]
 30%
0: {'loss': 0.5444, 'grad_norm': 1.5082820937371997, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.18}
0: |███       | 903/2984 [2:37:02<5:59:20, 10.36s/it]
 30%|███       | 904/2984 [2:37:12<5:59:54, 10.38s/it]
 30%|███       | 905/2984 [2:37:23<5:59:21, 10.37s/it]
 30%|███       | 906/2984 [2:37:33<5:58:59, 10.37s/it]
 30%|███       | 907/2984 [2:37:43<5:58:40, 10.36s/it]
 30%|███       | 908/2984 [2:37:54<5:58:54, 10.37s/it]
 30%|███       | 909/2984 [2:38:04<5:58:42, 10.37s/it]
 30%|███       | 910/2984 [2:38:15<5:59:10, 10.39s/it]
                                                      

 30%|███       | 910/2984 [2:38:15<5:59:10, 10.39s/it]
 31%|███       | 911/2984 [2:38:25<6:00:25, 10.43s/it]
 31%|███       | 912/2984 [2:38:35<6:00:06, 10.43s/it]
 31%|███       | 913/2984 [2:38:46<5:58:35, 10.39s/it]
 31%|███       | 914/2984 [2:38:56<5:58:53, 10.40s/it]
 31%|███       | 915/2984 [2:39:07<5:57:37, 10.37s/it]
 31%|███       | 916/2984 [2:39:17<5:56:50, 10.35s/it]
 31%|███       | 917/2984 [2:39:27<5:55:33, 10.32s/
0: {'loss': 0.5441, 'grad_norm': 0.9955405610962413, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.18}
0: {'loss': 0.5518, 'grad_norm': 0.980681521807772, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.19}
0: it]
 31%|███       | 918/2984 [2:39:37<5:53:46, 10.27s/it]
 31%|███       | 919/2984 [2:39:48<5:54:20, 10.30s/it]
 31%|███       | 920/2984 [2:39:58<5:54:41, 10.31s/it]
                                                      

 31%|███       | 920/2984 [2:39:58<5:54:41, 10.31s/it]
 31%|███       | 921/2984 [2:40:08<5:53:44, 10.29s/it]
 31%|███       | 922/2984 [2:40:18<5:53:14, 10.28s/it]
 31%|███       | 923/2984 [2:40:29<5:52:12, 10.25s/it]
 31%|███       | 924/2984 [2:40:39<5:51:57, 10.25s/it]
 31%|███       | 925/2984 [2:40:49<5:51:25, 10.24s/it]
 31%|███       | 926/2984 [2:40:59<5:51:32, 10.25s/it]
 31%|███       | 927/2984 [2:41:10<5:52:12, 10.27s/it]
 31%|███       | 928/2984 [2:41:20<5:51:47, 10.27s/it]
 31%|███       | 929/2984 [2:41:30<5:51:25, 10.26s/it]
 31%|███       | 930/2984 [2:41:40<5:49:34, 10.21s/it]
                                                      

 31%|███       | 930/2984 [2:41:40<5:49:34, 10.2
0: {'loss': 0.5497, 'grad_norm': 1.0038551602718155, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.19}
0: 1s/it]
 31%|███       | 931/2984 [2:41:50<5:48:56, 10.20s/it]
 31%|███       | 932/2984 [2:42:01<5:48:36, 10.19s/it]
 31%|███▏      | 933/2984 [2:42:11<5:48:40, 10.20s/it]
 31%|███▏      | 934/2984 [2:42:21<5:51:25, 10.29s/it]
 31%|███▏      | 935/2984 [2:42:32<5:52:11, 10.31s/it]
 31%|███▏      | 936/2984 [2:42:42<5:50:31, 10.27s/it]
 31%|███▏      | 937/2984 [2:42:52<5:48:59, 10.23s/it]
 31%|███▏      | 938/2984 [2:43:02<5:48:46, 10.23s/it]
 31%|███▏      | 939/2984 [2:43:12<5:47:12, 10.19s/it]
 32%|███▏      | 940/2984 [2:43:22<5:46:22, 10.17s/it]
                                                      

 32%|███▏      | 940/2984 [2:43:22<5:46:22, 10.17s/it]
 32%|███▏      | 941/2984 [2:43:33<5:46:41, 10.18s/it]
 32%|███▏      | 942/2984 [2:43:43<5:46:23, 10.18s/it]
 32%|███▏      | 943/2984 [2:43:53<5:46:20, 10.18s/it]
 32%|███▏      | 944/2984 [2:44:03<5:47:38, 10.22s/it]
 32%|███▏   
0: {'loss': 0.5331, 'grad_norm': 0.9721417813772719, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.19}
0:    | 945/2984 [2:44:14<5:50:14, 10.31s/it]
 32%|███▏      | 946/2984 [2:44:24<5:51:39, 10.35s/it]
 32%|███▏      | 947/2984 [2:44:35<5:52:14, 10.38s/it]
 32%|███▏      | 948/2984 [2:44:45<5:50:38, 10.33s/it]
 32%|███▏      | 949/2984 [2:44:55<5:49:39, 10.31s/it]
 32%|███▏      | 950/2984 [2:45:05<5:48:43, 10.29s/it]
                                                      

 32%|███▏      | 950/2984 [2:45:05<5:48:43, 10.29s/it]
 32%|███▏      | 951/2984 [2:45:16<5:49:03, 10.30s/it]
 32%|███▏      | 952/2984 [2:45:26<5:48:33, 10.29s/it]
 32%|███▏      | 953/2984 [2:45:36<5:47:30, 10.27s/it]
 32%|███▏      | 954/2984 [2:45:46<5:46:24, 10.24s/it]
 32%|███▏      | 955/2984 [2:45:57<5:46:21, 10.24s/it]
 32%|███▏      | 956/2984 [2:46:07<5:46:22, 10.25s/it]
 32%|███▏      | 957/2984 [2:46:17<5:46:30, 10.26s/it]
 32%|███▏      | 958/2984 [2:46:28<5:48:47, 10.33s/it]
 32%|███▏      | 959/2984 [2:46:38<
0: {'loss': 0.5478, 'grad_norm': 0.9604901534289484, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.19}
0: {'loss': 0.538, 'grad_norm': 0.9160467448830371, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.2}
0: 5:49:09, 10.35s/it]
 32%|███▏      | 960/2984 [2:46:48<5:47:55, 10.31s/it]
                                                      

 32%|███▏      | 960/2984 [2:46:48<5:47:55, 10.31s/it]
 32%|███▏      | 961/2984 [2:46:59<5:48:20, 10.33s/it]
 32%|███▏      | 962/2984 [2:47:09<5:47:57, 10.33s/it]
 32%|███▏      | 963/2984 [2:47:19<5:47:52, 10.33s/it]
 32%|███▏      | 964/2984 [2:47:30<5:47:45, 10.33s/it]
 32%|███▏      | 965/2984 [2:47:40<5:47:20, 10.32s/it]
 32%|███▏      | 966/2984 [2:47:50<5:47:13, 10.32s/it]
 32%|███▏      | 967/2984 [2:48:01<5:46:34, 10.31s/it]
 32%|███▏      | 968/2984 [2:48:11<5:46:45, 10.32s/it]
 32%|███▏      | 969/2984 [2:48:21<5:47:13, 10.34s/it]
 33%|███▎      | 970/2984 [2:48:32<5:48:26, 10.38s/it]
                                                      

 33%|███▎      | 970/2984 [2:48:32<5:48:26, 10.38s/it]
 33%|███▎      | 971/2984 [2:48:43<5:51:48, 10.49s/it]
 33%|█�
0: {'loss': 0.5487, 'grad_norm': 0.8670551443461478, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.2}
0: �█▎      | 972/2984 [2:48:53<5:54:31, 10.57s/it]
 33%|███▎      | 973/2984 [2:49:04<5:54:51, 10.59s/it]
 33%|███▎      | 974/2984 [2:49:14<5:53:08, 10.54s/it]
 33%|███▎      | 975/2984 [2:49:25<5:51:46, 10.51s/it]
 33%|███▎      | 976/2984 [2:49:35<5:50:29, 10.47s/it]
 33%|███▎      | 977/2984 [2:49:46<5:50:00, 10.46s/it]
 33%|███▎      | 978/2984 [2:49:56<5:49:58, 10.47s/it]
 33%|███▎      | 979/2984 [2:50:07<5:49:16, 10.45s/it]
 33%|███▎      | 980/2984 [2:50:17<5:48:06, 10.42s/it]
                                                      

 33%|███▎      | 980/2984 [2:50:17<5:48:06, 10.42s/it]
 33%|███▎      | 981/2984 [2:50:27<5:46:48, 10.39s/it]
 33%|███▎      | 982/2984 [2:50:37<5:45:11, 10.35s/it]
 33%|███▎      | 983/2984 [2:50:48<5:44:22, 10.33s/it]
 33%|███▎      | 984/2984 [2:50:58<5:44:01, 10.32s/it]
 33%|███▎      | 985/2984 [2:51:08<5:43:39, 10.31s/it]
 33%|███▎      | 986/2984
0: {'loss': 0.5536, 'grad_norm': 0.9004375539467191, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.2}
0:  [2:51:19<5:43:19, 10.31s/it]
 33%|███▎      | 987/2984 [2:51:29<5:42:37, 10.29s/it]
 33%|███▎      | 988/2984 [2:51:39<5:43:13, 10.32s/it]
 33%|███▎      | 989/2984 [2:51:50<5:42:46, 10.31s/it]
 33%|███▎      | 990/2984 [2:52:00<5:41:41, 10.28s/it]
                                                      

 33%|███▎      | 990/2984 [2:52:00<5:41:41, 10.28s/it]
 33%|███▎      | 991/2984 [2:52:10<5:41:39, 10.29s/it]
 33%|███▎      | 992/2984 [2:52:20<5:41:16, 10.28s/it]
 33%|███▎      | 993/2984 [2:52:31<5:41:05, 10.28s/it]
 33%|███▎      | 994/2984 [2:52:41<5:41:13, 10.29s/it]
 33%|███▎      | 995/2984 [2:52:51<5:41:28, 10.30s/it]
 33%|███▎      | 996/2984 [2:53:02<5:41:48, 10.32s/it]
 33%|███▎      | 997/2984 [2:53:12<5:41:33, 10.31s/it]
 33%|███▎      | 998/2984 [2:53:22<5:41:14, 10.31s/it]
 33%|███▎      | 999/2984 [2:53:32<5:40:52, 10.30s/it]
 34%|███▎      | 1000/2984 [2:53:43<5:39:48, 10.
0: {'loss': 0.5402, 'grad_norm': 0.9663089748689826, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.2}
0: {'loss': 0.549, 'grad_norm': 1.22820649301232, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.2}
0: 28s/it]
                                                       

 34%|███▎      | 1000/2984 [2:53:43<5:39:48, 10.28s/it]
 34%|███▎      | 1001/2984 [2:53:53<5:39:56, 10.29s/it]
 34%|███▎      | 1002/2984 [2:54:03<5:40:00, 10.29s/it]
 34%|███▎      | 1003/2984 [2:54:14<5:39:39, 10.29s/it]
 34%|███▎      | 1004/2984 [2:54:24<5:39:33, 10.29s/it]
 34%|███▎      | 1005/2984 [2:54:34<5:39:37, 10.30s/it]
 34%|███▎      | 1006/2984 [2:54:44<5:38:58, 10.28s/it]
 34%|███▎      | 1007/2984 [2:54:55<5:38:13, 10.26s/it]
 34%|███▍      | 1008/2984 [2:55:05<5:38:01, 10.26s/it]
 34%|███▍      | 1009/2984 [2:55:15<5:37:45, 10.26s/it]
 34%|███▍      | 1010/2984 [2:55:25<5:37:21, 10.25s/it]
                                                       

 34%|███▍      | 1010/2984 [2:55:25<5:37:21, 10.25s/it]
 34%|███▍      | 1011/2984 [2:55:36<5:37:07, 10.25s/it]
 34%|███▍      | 1012/2984 [2:55:46<5:38:04, 10.29s/it]
 34%|�
0: {'loss': 0.5405, 'grad_norm': 1.0554634285468756, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.21}
0: ��██▍      | 1013/2984 [2:55:56<5:39:28, 10.33s/it]
 34%|███▍      | 1014/2984 [2:56:07<5:39:14, 10.33s/it]
 34%|███▍      | 1015/2984 [2:56:17<5:38:46, 10.32s/it]
 34%|███▍      | 1016/2984 [2:56:27<5:38:03, 10.31s/it]
 34%|███▍      | 1017/2984 [2:56:38<5:36:58, 10.28s/it]
 34%|███▍      | 1018/2984 [2:56:48<5:37:24, 10.30s/it]
 34%|███▍      | 1019/2984 [2:56:58<5:36:40, 10.28s/it]
 34%|███▍      | 1020/2984 [2:57:08<5:36:17, 10.27s/it]
                                                       

 34%|███▍      | 1020/2984 [2:57:08<5:36:17, 10.27s/it]
 34%|███▍      | 1021/2984 [2:57:19<5:35:44, 10.26s/it]
 34%|███▍      | 1022/2984 [2:57:29<5:35:05, 10.25s/it]
 34%|███▍      | 1023/2984 [2:57:39<5:34:37, 10.24s/it]
 34%|███▍      | 1024/2984 [2:57:49<5:33:46, 10.22s/it]
 34%|███▍      | 1025/2984 [2:57:59<5:31:27, 10.15s/it]
 34%|███▍      | 1026/2984 [2:58:09<5:30:12, 10.12s/it]
 34%|██�
0: {'loss': 0.5314, 'grad_norm': 0.9867838355131848, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.21}
0: �▍      | 1027/2984 [2:58:19<5:29:43, 10.11s/it]
 34%|███▍      | 1028/2984 [2:58:30<5:29:43, 10.11s/it]
 34%|███▍      | 1029/2984 [2:58:40<5:29:35, 10.12s/it]
 35%|███▍      | 1030/2984 [2:58:50<5:29:23, 10.11s/it]
                                                       

 35%|███▍      | 1030/2984 [2:58:50<5:29:23, 10.11s/it]
 35%|███▍      | 1031/2984 [2:59:00<5:29:18, 10.12s/it]
 35%|███▍      | 1032/2984 [2:59:10<5:29:37, 10.13s/it]
 35%|███▍      | 1033/2984 [2:59:20<5:30:17, 10.16s/it]
 35%|███▍      | 1034/2984 [2:59:30<5:30:45, 10.18s/it]
 35%|███▍      | 1035/2984 [2:59:41<5:30:28, 10.17s/it]
 35%|███▍      | 1036/2984 [2:59:51<5:30:12, 10.17s/it]
 35%|███▍      | 1037/2984 [3:00:01<5:29:43, 10.16s/it]
 35%|███▍      | 1038/2984 [3:00:11<5:30:49, 10.20s/it]
 35%|███▍      | 1039/2984 [3:00:21<5:29:39, 10.17s/it]
 35%|███▍      | 1040/2984 [3:00:31<5:29:06, 10.16s/it]
                    
0: {'loss': 0.5408, 'grad_norm': 0.9118371620046222, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.21}
0: {'loss': 0.5418, 'grad_norm': 0.9841060704735403, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.21}
0:                                    

 35%|███▍      | 1040/2984 [3:00:31<5:29:06, 10.16s/it]
 35%|███▍      | 1041/2984 [3:00:42<5:28:52, 10.16s/it]
 35%|███▍      | 1042/2984 [3:00:52<5:28:08, 10.14s/it]
 35%|███▍      | 1043/2984 [3:01:02<5:28:41, 10.16s/it]
 35%|███▍      | 1044/2984 [3:01:12<5:28:12, 10.15s/it]
 35%|███▌      | 1045/2984 [3:01:22<5:27:35, 10.14s/it]
 35%|███▌      | 1046/2984 [3:01:32<5:26:31, 10.11s/it]
 35%|███▌      | 1047/2984 [3:01:42<5:24:52, 10.06s/it]
 35%|███▌      | 1048/2984 [3:01:52<5:24:06, 10.04s/it]
 35%|███▌      | 1049/2984 [3:02:02<5:23:50, 10.04s/it]
 35%|███▌      | 1050/2984 [3:02:12<5:23:29, 10.04s/it]
                                                       

 35%|███▌      | 1050/2984 [3:02:12<5:23:29, 10.04s/it]
 35%|███▌      | 1051/2984 [3:02:22<5:23:11, 10.03s/it]
 35%|███▌      | 1052/2984 [3:02:32<5:23:24, 10.04s/it]
 35%|███▌      | 1053/2984
0: {'loss': 0.5423, 'grad_norm': 1.0209572150813044, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.21}
0:  [3:02:43<5:25:50, 10.12s/it]
 35%|███▌      | 1054/2984 [3:02:53<5:27:15, 10.17s/it]
 35%|███▌      | 1055/2984 [3:03:03<5:25:20, 10.12s/it]
 35%|███▌      | 1056/2984 [3:03:13<5:24:22, 10.09s/it]
 35%|███▌      | 1057/2984 [3:03:23<5:24:16, 10.10s/it]
 35%|███▌      | 1058/2984 [3:03:33<5:24:04, 10.10s/it]
 35%|███▌      | 1059/2984 [3:03:43<5:24:01, 10.10s/it]
 36%|███▌      | 1060/2984 [3:03:53<5:23:58, 10.10s/it]
                                                       

 36%|███▌      | 1060/2984 [3:03:53<5:23:58, 10.10s/it]
 36%|███▌      | 1061/2984 [3:04:04<5:24:13, 10.12s/it]
 36%|███▌      | 1062/2984 [3:04:14<5:23:46, 10.11s/it]
 36%|███▌      | 1063/2984 [3:04:24<5:23:37, 10.11s/it]
 36%|███▌      | 1064/2984 [3:04:34<5:24:06, 10.13s/it]
 36%|███▌      | 1065/2984 [3:04:44<5:23:53, 10.13s/it]
 36%|███▌      | 1066/2984 [3:04:54<5:23:54, 10.13s/it]
 36%|███▌      | 1067/2984 [3:05:
0: {'loss': 0.5396, 'grad_norm': 0.9738323967727966, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.22}
0: 04<5:23:50, 10.14s/it]
 36%|███▌      | 1068/2984 [3:05:14<5:23:53, 10.14s/it]
 36%|███▌      | 1069/2984 [3:05:25<5:23:13, 10.13s/it]
 36%|███▌      | 1070/2984 [3:05:35<5:22:35, 10.11s/it]
                                                       

 36%|███▌      | 1070/2984 [3:05:35<5:22:35, 10.11s/it]
 36%|███▌      | 1071/2984 [3:05:45<5:22:51, 10.13s/it]
 36%|███▌      | 1072/2984 [3:05:55<5:22:26, 10.12s/it]
 36%|███▌      | 1073/2984 [3:06:05<5:21:50, 10.11s/it]
 36%|███▌      | 1074/2984 [3:06:15<5:21:43, 10.11s/it]
 36%|███▌      | 1075/2984 [3:06:25<5:22:24, 10.13s/it]
 36%|███▌      | 1076/2984 [3:06:35<5:22:37, 10.15s/it]
 36%|███▌      | 1077/2984 [3:06:46<5:23:29, 10.18s/it]
 36%|███▌      | 1078/2984 [3:06:56<5:23:56, 10.20s/it]
 36%|███▌      | 1079/2984 [3:07:06<5:25:52, 10.26s/it]
 36%|███▌      | 1080/2984 [3:07:17<5:28:12, 10.34s/it]
                                                
0: {'loss': 0.5365, 'grad_norm': 0.8406050718002787, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.22}
0: {'loss': 0.5448, 'grad_norm': 0.9357258780546838, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.22}
0:        

 36%|███▌      | 1080/2984 [3:07:17<5:28:12, 10.34s/it]
 36%|███▌      | 1081/2984 [3:07:27<5:27:39, 10.33s/it]
 36%|███▋      | 1082/2984 [3:07:37<5:26:52, 10.31s/it]
 36%|███▋      | 1083/2984 [3:07:48<5:26:08, 10.29s/it]
 36%|███▋      | 1084/2984 [3:07:58<5:26:51, 10.32s/it]
 36%|███▋      | 1085/2984 [3:08:08<5:26:00, 10.30s/it]
 36%|███▋      | 1086/2984 [3:08:19<5:25:55, 10.30s/it]
 36%|███▋      | 1087/2984 [3:08:29<5:23:34, 10.23s/it]
 36%|███▋      | 1088/2984 [3:08:39<5:22:18, 10.20s/it]
 36%|███▋      | 1089/2984 [3:08:49<5:20:24, 10.14s/it]
 37%|███▋      | 1090/2984 [3:08:59<5:18:31, 10.09s/it]
                                                       

 37%|███▋      | 1090/2984 [3:08:59<5:18:31, 10.09s/it]
 37%|███▋      | 1091/2984 [3:09:09<5:18:58, 10.11s/it]
 37%|███▋      | 1092/2984 [3:09:19<5:19:36, 10.14s/it]
 37%|███▋      | 1093/2984 [3:09:29<5:20:35, 10.17s/it
0: {'loss': 0.53, 'grad_norm': 0.8657176177142817, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.22}
0: ]
 37%|███▋      | 1094/2984 [3:09:40<5:21:12, 10.20s/it]
 37%|███▋      | 1095/2984 [3:09:50<5:21:35, 10.21s/it]
 37%|███▋      | 1096/2984 [3:10:00<5:21:53, 10.23s/it]
 37%|███▋      | 1097/2984 [3:10:10<5:21:25, 10.22s/it]
 37%|███▋      | 1098/2984 [3:10:20<5:19:38, 10.17s/it]
 37%|███▋      | 1099/2984 [3:10:31<5:18:22, 10.13s/it]
 37%|███▋      | 1100/2984 [3:10:41<5:18:00, 10.13s/it]
                                                       

 37%|███▋      | 1100/2984 [3:10:41<5:18:00, 10.13s/it]
 37%|███▋      | 1101/2984 [3:10:51<5:18:17, 10.14s/it]
 37%|███▋      | 1102/2984 [3:11:01<5:18:31, 10.15s/it]
 37%|███▋      | 1103/2984 [3:11:11<5:18:46, 10.17s/it]
 37%|███▋      | 1104/2984 [3:11:21<5:18:29, 10.16s/it]
 37%|███▋      | 1105/2984 [3:11:31<5:17:42, 10.15s/it]
 37%|███▋      | 1106/2984 [3:11:42<5:16:53, 10.12s/it]
 37%|███▋      | 1107/2984 [3:11:52<5:15:37, 10.09s/it]
 37%|
0: {'loss': 0.5357, 'grad_norm': 1.0338483170384627, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.22}
0: {'loss': 0.5234, 'grad_norm': 1.0956829717436956, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.23}
0: ███▋      | 1108/2984 [3:12:02<5:14:54, 10.07s/it]
 37%|███▋      | 1109/2984 [3:12:12<5:14:59, 10.08s/it]
 37%|███▋      | 1110/2984 [3:12:22<5:15:21, 10.10s/it]
                                                       

 37%|███▋      | 1110/2984 [3:12:22<5:15:21, 10.10s/it]
 37%|███▋      | 1111/2984 [3:12:32<5:16:48, 10.15s/it]
 37%|███▋      | 1112/2984 [3:12:42<5:18:46, 10.22s/it]
 37%|███▋      | 1113/2984 [3:12:53<5:18:50, 10.22s/it]
 37%|███▋      | 1114/2984 [3:13:03<5:18:34, 10.22s/it]
 37%|███▋      | 1115/2984 [3:13:13<5:17:41, 10.20s/it]
 37%|███▋      | 1116/2984 [3:13:23<5:15:56, 10.15s/it]
 37%|███▋      | 1117/2984 [3:13:33<5:15:59, 10.15s/it]
 37%|███▋      | 1118/2984 [3:13:43<5:15:30, 10.14s/it]
 38%|███▊      | 1119/2984 [3:13:53<5:13:31, 10.09s/it]
 38%|███▊      | 1120/2984 [3:14:03<5:12:12, 10.05s/it]
                                                       

 38%|███▊  
0: {'loss': 0.5338, 'grad_norm': 0.9179581197865339, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.23}
0:     | 1120/2984 [3:14:03<5:12:12, 10.05s/it]
 38%|███▊      | 1121/2984 [3:14:13<5:13:35, 10.10s/it]
 38%|███▊      | 1122/2984 [3:14:24<5:15:16, 10.16s/it]
 38%|███▊      | 1123/2984 [3:14:34<5:14:15, 10.13s/it]
 38%|███▊      | 1124/2984 [3:14:44<5:12:27, 10.08s/it]
 38%|███▊      | 1125/2984 [3:14:54<5:11:05, 10.04s/it]
 38%|███▊      | 1126/2984 [3:15:04<5:11:53, 10.07s/it]
 38%|███▊      | 1127/2984 [3:15:14<5:14:03, 10.15s/it]
 38%|███▊      | 1128/2984 [3:15:25<5:15:28, 10.20s/it]
 38%|███▊      | 1129/2984 [3:15:35<5:17:15, 10.26s/it]
 38%|███▊      | 1130/2984 [3:15:45<5:16:41, 10.25s/it]
                                                       

 38%|███▊      | 1130/2984 [3:15:45<5:16:41, 10.25s/it]
 38%|███▊      | 1131/2984 [3:15:56<5:17:39, 10.29s/it]
 38%|███▊      | 1132/2984 [3:16:06<5:18:32, 10.32s/it]
 38%|███▊      | 1133/2984 [3:16:16<5:18:00, 10.31s/it]
 38%|███▊      | 1
0: {'loss': 0.5258, 'grad_norm': 1.0038197020340918, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.23}
0: 134/2984 [3:16:26<5:16:51, 10.28s/it]
 38%|███▊      | 1135/2984 [3:16:37<5:16:15, 10.26s/it]
 38%|███▊      | 1136/2984 [3:16:47<5:15:23, 10.24s/it]
 38%|███▊      | 1137/2984 [3:16:57<5:15:11, 10.24s/it]
 38%|███▊      | 1138/2984 [3:17:07<5:14:52, 10.23s/it]
 38%|███▊      | 1139/2984 [3:17:18<5:14:39, 10.23s/it]
 38%|███▊      | 1140/2984 [3:17:28<5:15:09, 10.25s/it]
                                                       

 38%|███▊      | 1140/2984 [3:17:28<5:15:09, 10.25s/it]
 38%|███▊      | 1141/2984 [3:17:38<5:15:55, 10.28s/it]
 38%|███▊      | 1142/2984 [3:17:48<5:15:50, 10.29s/it]
 38%|███▊      | 1143/2984 [3:17:59<5:15:43, 10.29s/it]
 38%|███▊      | 1144/2984 [3:18:09<5:14:59, 10.27s/it]
 38%|███▊      | 1145/2984 [3:18:19<5:14:50, 10.27s/it]
 38%|███▊      | 1146/2984 [3:18:30<5:14:47, 10.28s/it]
 38%|███▊      | 1147/2984 [3:18:40<5:14:33, 10.27s/it]
 38%|███▊      | 1148/298
0: {'loss': 0.5362, 'grad_norm': 0.9919510179108267, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.23}
0: {'loss': 0.5371, 'grad_norm': 1.1651390476841703, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.23}
0: 4 [3:18:50<5:13:27, 10.24s/it]
 39%|███▊      | 1149/2984 [3:19:00<5:12:30, 10.22s/it]
 39%|███▊      | 1150/2984 [3:19:10<5:12:00, 10.21s/it]
                                                       

 39%|███▊      | 1150/2984 [3:19:10<5:12:00, 10.21s/it]
 39%|███▊      | 1151/2984 [3:19:21<5:12:02, 10.21s/it]
 39%|███▊      | 1152/2984 [3:19:31<5:12:41, 10.24s/it]
 39%|███▊      | 1153/2984 [3:19:41<5:11:49, 10.22s/it]
 39%|███▊      | 1154/2984 [3:19:51<5:10:15, 10.17s/it]
 39%|███▊      | 1155/2984 [3:20:01<5:10:03, 10.17s/it]
 39%|███▊      | 1156/2984 [3:20:11<5:09:40, 10.16s/it]
 39%|███▉      | 1157/2984 [3:20:22<5:09:27, 10.16s/it]
 39%|███▉      | 1158/2984 [3:20:32<5:09:28, 10.17s/it]
 39%|███▉      | 1159/2984 [3:20:42<5:10:40, 10.21s/it]
 39%|███▉      | 1160/2984 [3:20:52<5:10:30, 10.21s/it]
                                                       

 39%|███▉      | 1160/2984 [3:20:52<5:1
0: {'loss': 0.5469, 'grad_norm': 1.05185806943696, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.24}
0: 0:30, 10.21s/it]
 39%|███▉      | 1161/2984 [3:21:03<5:10:17, 10.21s/it]
 39%|███▉      | 1162/2984 [3:21:13<5:10:37, 10.23s/it]
 39%|███▉      | 1163/2984 [3:21:23<5:12:04, 10.28s/it]
 39%|███▉      | 1164/2984 [3:21:33<5:11:50, 10.28s/it]
 39%|███▉      | 1165/2984 [3:21:44<5:10:36, 10.25s/it]
 39%|███▉      | 1166/2984 [3:21:54<5:09:49, 10.23s/it]
 39%|███▉      | 1167/2984 [3:22:04<5:09:07, 10.21s/it]
 39%|███▉      | 1168/2984 [3:22:14<5:08:28, 10.19s/it]
 39%|███▉      | 1169/2984 [3:22:24<5:07:44, 10.17s/it]
 39%|███▉      | 1170/2984 [3:22:34<5:06:45, 10.15s/it]
                                                       

 39%|███▉      | 1170/2984 [3:22:34<5:06:45, 10.15s/it]
 39%|███▉      | 1171/2984 [3:22:44<5:06:29, 10.14s/it]
 39%|███▉      | 1172/2984 [3:22:55<5:05:59, 10.13s/it]
 39%|███▉      | 1173/2984 [3:23:05<5:07:40, 10.19s/it]
 39%|███▉      | 1174/2984 [3:23:15<5:09:12, 1
0: {'loss': 0.5396, 'grad_norm': 1.2006348698206446, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.24}
0: 0.25s/it]
 39%|███▉      | 1175/2984 [3:23:26<5:09:34, 10.27s/it]
 39%|███▉      | 1176/2984 [3:23:36<5:09:10, 10.26s/it]
 39%|███▉      | 1177/2984 [3:23:46<5:08:55, 10.26s/it]
 39%|███▉      | 1178/2984 [3:23:56<5:08:32, 10.25s/it]
 40%|███▉      | 1179/2984 [3:24:07<5:08:11, 10.24s/it]
 40%|███▉      | 1180/2984 [3:24:17<5:08:07, 10.25s/it]
                                                       

 40%|███▉      | 1180/2984 [3:24:17<5:08:07, 10.25s/it]
 40%|███▉      | 1181/2984 [3:24:27<5:09:56, 10.31s/it]
 40%|███▉      | 1182/2984 [3:24:38<5:08:51, 10.28s/it]
 40%|███▉      | 1183/2984 [3:24:48<5:07:44, 10.25s/it]
 40%|███▉      | 1184/2984 [3:24:58<5:05:30, 10.18s/it]
 40%|███▉      | 1185/2984 [3:25:08<5:03:53, 10.14s/it]
 40%|███▉      | 1186/2984 [3:25:18<5:01:54, 10.07s/it]
 40%|███▉      | 1187/2984 [3:25:28<5:00:30, 10.03s/it]
 40%|███▉      | 1188/2984 [3:25:38<4:59:34, 10.01s/i
0: {'loss': 0.5329, 'grad_norm': 1.5522943087073797, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.24}
0: {'loss': 0.5198, 'grad_norm': 0.8693814932673896, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.24}
0: t]
 40%|███▉      | 1189/2984 [3:25:48<5:00:31, 10.05s/it]
 40%|███▉      | 1190/2984 [3:25:58<5:00:01, 10.03s/it]
                                                       

 40%|███▉      | 1190/2984 [3:25:58<5:00:01, 10.03s/it]
 40%|███▉      | 1191/2984 [3:26:08<5:00:54, 10.07s/it]
 40%|███▉      | 1192/2984 [3:26:18<5:01:52, 10.11s/it]
 40%|███▉      | 1193/2984 [3:26:28<5:02:31, 10.13s/it]
 40%|████      | 1194/2984 [3:26:38<5:03:13, 10.16s/it]
 40%|████      | 1195/2984 [3:26:49<5:04:01, 10.20s/it]
 40%|████      | 1196/2984 [3:26:59<5:04:38, 10.22s/it]
 40%|████      | 1197/2984 [3:27:09<5:04:24, 10.22s/it]
 40%|████      | 1198/2984 [3:27:19<5:04:24, 10.23s/it]
 40%|████      | 1199/2984 [3:27:30<5:05:54, 10.28s/it]
 40%|████      | 1200/2984 [3:27:40<5:07:03, 10.33s/it]
                                                       

 40%|████      | 1200/2984 [3:27:40<5:07:03, 10.33s/it]
 40%|██
0: {'loss': 0.5351, 'grad_norm': 1.0527234014183586, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.24}
0: ██      | 1201/2984 [3:27:51<5:07:56, 10.36s/it]
 40%|████      | 1202/2984 [3:28:01<5:08:06, 10.37s/it]
 40%|████      | 1203/2984 [3:28:12<5:09:10, 10.42s/it]
 40%|████      | 1204/2984 [3:28:22<5:10:18, 10.46s/it]
 40%|████      | 1205/2984 [3:28:33<5:11:31, 10.51s/it]
 40%|████      | 1206/2984 [3:28:44<5:13:21, 10.57s/it]
 40%|████      | 1207/2984 [3:28:54<5:13:25, 10.58s/it]
 40%|████      | 1208/2984 [3:29:05<5:12:29, 10.56s/it]
 41%|████      | 1209/2984 [3:29:15<5:11:41, 10.54s/it]
 41%|████      | 1210/2984 [3:29:26<5:11:50, 10.55s/it]
                                                       

 41%|████      | 1210/2984 [3:29:26<5:11:50, 10.55s/it]
 41%|████      | 1211/2984 [3:29:36<5:12:02, 10.56s/it]
 41%|████      | 1212/2984 [3:29:47<5:11:45, 10.56s/it]
 41%|████      | 1213/2984 [3:29:57<5:11:30, 10.55s/it]
 41%|████      | 1214/2984 [3:30:08<5:11:15, 10.55s/it]
 41%|████ 
0: {'loss': 0.5429, 'grad_norm': 0.9394647114284991, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.25}
0:      | 1215/2984 [3:30:18<5:10:20, 10.53s/it]
 41%|████      | 1216/2984 [3:30:29<5:07:53, 10.45s/it]
 41%|████      | 1217/2984 [3:30:39<5:05:59, 10.39s/it]
 41%|████      | 1218/2984 [3:30:49<5:03:47, 10.32s/it]
 41%|████      | 1219/2984 [3:30:59<5:02:14, 10.27s/it]
 41%|████      | 1220/2984 [3:31:09<5:00:41, 10.23s/it]
                                                       

 41%|████      | 1220/2984 [3:31:09<5:00:41, 10.23s/it]
 41%|████      | 1221/2984 [3:31:20<4:59:39, 10.20s/it]
 41%|████      | 1222/2984 [3:31:30<4:59:00, 10.18s/it]
 41%|████      | 1223/2984 [3:31:40<5:00:03, 10.22s/it]
 41%|████      | 1224/2984 [3:31:50<4:59:48, 10.22s/it]
 41%|████      | 1225/2984 [3:32:00<4:59:06, 10.20s/it]
 41%|████      | 1226/2984 [3:32:11<4:58:45, 10.20s/it]
 41%|████      | 1227/2984 [3:32:21<4:57:38, 10.16s/it]
 41%|████      | 1228/2984 [3:32:31<4:59:03, 10.22s/it]
 41%|████      | 
0: {'loss': 0.5318, 'grad_norm': 1.0046996217204216, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.25}
0: {'loss': 0.5375, 'grad_norm': 0.855216278062273, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.25}
0: 1229/2984 [3:32:41<4:59:50, 10.25s/it]
 41%|████      | 1230/2984 [3:32:51<4:58:47, 10.22s/it]
                                                       

 41%|████      | 1230/2984 [3:32:52<4:58:47, 10.22s/it]
 41%|████▏     | 1231/2984 [3:33:02<4:57:50, 10.19s/it]
 41%|████▏     | 1232/2984 [3:33:12<4:57:03, 10.17s/it]
 41%|████▏     | 1233/2984 [3:33:22<4:56:48, 10.17s/it]
 41%|████▏     | 1234/2984 [3:33:32<4:57:16, 10.19s/it]
 41%|████▏     | 1235/2984 [3:33:42<4:56:05, 10.16s/it]
 41%|████▏     | 1236/2984 [3:33:52<4:55:28, 10.14s/it]
 41%|████▏     | 1237/2984 [3:34:02<4:54:58, 10.13s/it]
 41%|████▏     | 1238/2984 [3:34:13<4:54:23, 10.12s/it]
 42%|████▏     | 1239/2984 [3:34:23<4:53:40, 10.10s/it]
 42%|████▏     | 1240/2984 [3:34:33<4:52:50, 10.07s/it]
                                                       

 42%|████▏     | 1240/2984 [3:34:33<4:52:50, 10.07s/it]
 42%|████
0: {'loss': 0.5287, 'grad_norm': 0.8453346776087125, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.25}
0: ▏     | 1241/2984 [3:34:43<4:52:09, 10.06s/it]
 42%|████▏     | 1242/2984 [3:34:53<4:51:36, 10.04s/it]
 42%|████▏     | 1243/2984 [3:35:03<4:51:50, 10.06s/it]
 42%|████▏     | 1244/2984 [3:35:13<4:52:10, 10.08s/it]
 42%|████▏     | 1245/2984 [3:35:23<4:54:12, 10.15s/it]
 42%|████▏     | 1246/2984 [3:35:33<4:55:30, 10.20s/it]
 42%|████▏     | 1247/2984 [3:35:44<4:56:12, 10.23s/it]
 42%|████▏     | 1248/2984 [3:35:54<4:59:24, 10.35s/it]
 42%|████▏     | 1249/2984 [3:36:05<4:59:33, 10.36s/it]
 42%|████▏     | 1250/2984 [3:36:15<5:00:45, 10.41s/it]
                                                       

 42%|████▏     | 1250/2984 [3:36:15<5:00:45, 10.41s/it]
 42%|████▏     | 1251/2984 [3:36:26<5:01:26, 10.44s/it]
 42%|████▏     | 1252/2984 [3:36:36<5:01:51, 10.46s/it]
 42%|████▏     | 1253/2984 [3:36:46<4:58:58, 10.36s/it]
 42%|████▏     | 1254/2984 [3:36:57<4:58:24, 10.35
0: {'loss': 0.5366, 'grad_norm': 1.1002500243253832, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.25}
0: s/it]
 42%|████▏     | 1255/2984 [3:37:07<4:58:34, 10.36s/it]
 42%|████▏     | 1256/2984 [3:37:18<4:58:46, 10.37s/it]
 42%|████▏     | 1257/2984 [3:37:28<4:58:57, 10.39s/it]
 42%|████▏     | 1258/2984 [3:37:38<4:59:19, 10.41s/it]
 42%|████▏     | 1259/2984 [3:37:49<4:56:39, 10.32s/it]
 42%|████▏     | 1260/2984 [3:37:59<4:54:10, 10.24s/it]
                                                       

 42%|████▏     | 1260/2984 [3:37:59<4:54:10, 10.24s/it]
 42%|████▏     | 1261/2984 [3:38:09<4:52:29, 10.19s/it]
 42%|████▏     | 1262/2984 [3:38:19<4:51:51, 10.17s/it]
 42%|████▏     | 1263/2984 [3:38:29<4:51:24, 10.16s/it]
 42%|████▏     | 1264/2984 [3:38:39<4:50:19, 10.13s/it]
 42%|████▏     | 1265/2984 [3:38:49<4:50:17, 10.13s/it]
 42%|████▏     | 1266/2984 [3:38:59<4:49:16, 10.10s/it]
 42%|████▏     | 1267/2984 [3:39:09<4:51:01, 10.17s/it]
 42%|████▏     | 1268/2984 
0: {'loss': 0.5308, 'grad_norm': 0.9223099388539636, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.26}
0: {'loss': 0.5342, 'grad_norm': 0.9325296864033885, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.26}
0: [3:39:20<4:50:09, 10.15s/it]
 43%|████▎     | 1269/2984 [3:39:30<4:49:42, 10.14s/it]
 43%|████▎     | 1270/2984 [3:39:40<4:49:38, 10.14s/it]
                                                       

 43%|████▎     | 1270/2984 [3:39:40<4:49:38, 10.14s/it]
 43%|████▎     | 1271/2984 [3:39:50<4:50:07, 10.16s/it]
 43%|████▎     | 1272/2984 [3:40:00<4:49:15, 10.14s/it]
 43%|████▎     | 1273/2984 [3:40:10<4:48:24, 10.11s/it]
 43%|████▎     | 1274/2984 [3:40:20<4:48:14, 10.11s/it]
 43%|████▎     | 1275/2984 [3:40:31<4:49:18, 10.16s/it]
 43%|████▎     | 1276/2984 [3:40:41<4:49:57, 10.19s/it]
 43%|████▎     | 1277/2984 [3:40:51<4:48:34, 10.14s/it]
 43%|████▎     | 1278/2984 [3:41:01<4:47:53, 10.13s/it]
 43%|████▎     | 1279/2984 [3:41:11<4:47:40, 10.12s/it]
 43%|████▎     | 1280/2984 [3:41:21<4:48:05, 10.14s/it]
                                                       

 43%|████▎   
0: {'loss': 0.5409, 'grad_norm': 0.9870126133340061, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.26}
0:   | 1280/2984 [3:41:21<4:48:05, 10.14s/it]
 43%|████▎     | 1281/2984 [3:41:31<4:47:21, 10.12s/it]
 43%|████▎     | 1282/2984 [3:41:41<4:47:02, 10.12s/it]
 43%|████▎     | 1283/2984 [3:41:52<4:47:00, 10.12s/it]
 43%|████▎     | 1284/2984 [3:42:02<4:53:30, 10.36s/it]
 43%|████▎     | 1285/2984 [3:42:13<4:55:16, 10.43s/it]
 43%|████▎     | 1286/2984 [3:42:24<4:58:17, 10.54s/it]
 43%|████▎     | 1287/2984 [3:42:35<5:02:29, 10.70s/it]
 43%|████▎     | 1288/2984 [3:42:45<4:59:53, 10.61s/it]
 43%|████▎     | 1289/2984 [3:42:56<4:58:06, 10.55s/it]
 43%|████▎     | 1290/2984 [3:43:06<4:56:31, 10.50s/it]
                                                       

 43%|████▎     | 1290/2984 [3:43:06<4:56:31, 10.50s/it]
 43%|████▎     | 1291/2984 [3:43:17<4:58:43, 10.59s/it]
 43%|████▎     | 1292/2984 [3:43:28<5:00:39, 10.66s/it]
 43%|████▎     | 1293/2984 [3:43:38<4:58:16, 10.58s/it]
0: {'loss': 0.518, 'grad_norm': 1.0107821405083308, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.26}
0:  43%|████▎     | 1294/2984 [3:43:48<4:55:52, 10.50s/it]
 43%|████▎     | 1295/2984 [3:43:59<4:53:15, 10.42s/it]
 43%|████▎     | 1296/2984 [3:44:09<4:51:21, 10.36s/it]
 43%|████▎     | 1297/2984 [3:44:19<4:50:08, 10.32s/it]
 43%|████▎     | 1298/2984 [3:44:29<4:48:10, 10.26s/it]
 44%|████▎     | 1299/2984 [3:44:39<4:46:04, 10.19s/it]
 44%|████▎     | 1300/2984 [3:44:49<4:44:54, 10.15s/it]
                                                       

 44%|████▎     | 1300/2984 [3:44:49<4:44:54, 10.15s/it]
 44%|████▎     | 1301/2984 [3:45:00<4:45:45, 10.19s/it]
 44%|████▎     | 1302/2984 [3:45:10<4:46:56, 10.24s/it]
 44%|████▎     | 1303/2984 [3:45:20<4:46:50, 10.24s/it]
 44%|████▎     | 1304/2984 [3:45:30<4:45:44, 10.21s/it]
 44%|████▎     | 1305/2984 [3:45:40<4:44:52, 10.18s/it]
 44%|████▍     | 1306/2984 [3:45:51<4:45:12, 10.20s/it]
 44%|████▍     | 1307/2984 [3:46:
0: {'loss': 0.5184, 'grad_norm': 0.8806101651402196, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.26}
0: 01<4:44:55, 10.19s/it]
 44%|████▍     | 1308/2984 [3:46:11<4:44:40, 10.19s/it]
 44%|████▍     | 1309/2984 [3:46:21<4:43:40, 10.16s/it]
 44%|████▍     | 1310/2984 [3:46:31<4:43:21, 10.16s/it]
                                                       

 44%|████▍     | 1310/2984 [3:46:31<4:43:21, 10.16s/it]
 44%|████▍     | 1311/2984 [3:46:42<4:46:14, 10.27s/it]
 44%|████▍     | 1312/2984 [3:46:52<4:45:32, 10.25s/it]
 44%|████▍     | 1313/2984 [3:47:02<4:44:20, 10.21s/it]
 44%|████▍     | 1314/2984 [3:47:12<4:44:25, 10.22s/it]
 44%|████▍     | 1315/2984 [3:47:23<4:44:15, 10.22s/it]
 44%|████▍     | 1316/2984 [3:47:33<4:44:19, 10.23s/it]
 44%|████▍     | 1317/2984 [3:47:43<4:46:53, 10.33s/it]
 44%|████▍     | 1318/2984 [3:47:54<4:45:58, 10.30s/it]
 44%|████▍     | 1319/2984 [3:48:04<4:44:47, 10.26s/it]
 44%|████▍     | 1320/2984 [3:48:14<4:43:51, 10.24s/it]
                    
0: {'loss': 0.5308, 'grad_norm': 1.0076977777911411, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.27}
0: {'loss': 0.5291, 'grad_norm': 1.1197070378053398, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.27}
0:                                    

 44%|████▍     | 1320/2984 [3:48:14<4:43:51, 10.24s/it]
 44%|████▍     | 1321/2984 [3:48:24<4:42:04, 10.18s/it]
 44%|████▍     | 1322/2984 [3:48:34<4:41:28, 10.16s/it]
 44%|████▍     | 1323/2984 [3:48:44<4:41:28, 10.17s/it]
 44%|████▍     | 1324/2984 [3:48:54<4:40:51, 10.15s/it]
 44%|████▍     | 1325/2984 [3:49:05<4:40:37, 10.15s/it]
 44%|████▍     | 1326/2984 [3:49:15<4:40:54, 10.17s/it]
 44%|████▍     | 1327/2984 [3:49:25<4:41:03, 10.18s/it]
 45%|████▍     | 1328/2984 [3:49:35<4:41:00, 10.18s/it]
 45%|████▍     | 1329/2984 [3:49:45<4:41:12, 10.19s/it]
 45%|████▍     | 1330/2984 [3:49:56<4:40:12, 10.16s/it]
                                                       

 45%|████▍     | 1330/2984 [3:49:56<4:40:12, 10.16s/it]
 45%|████▍     | 1331/2984 [3:50:06<4:40:12, 10.17s/it]
 45%|████▍     | 1332/2984 [3:50:16<4:39:58, 10.17s/it]
 45%|�
0: {'loss': 0.5377, 'grad_norm': 1.1881984458279398, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.27}
0: ��███▍     | 1333/2984 [3:50:26<4:39:03, 10.14s/it]
 45%|████▍     | 1334/2984 [3:50:36<4:38:16, 10.12s/it]
 45%|████▍     | 1335/2984 [3:50:46<4:37:58, 10.11s/it]
 45%|████▍     | 1336/2984 [3:50:56<4:37:47, 10.11s/it]
 45%|████▍     | 1337/2984 [3:51:06<4:37:45, 10.12s/it]
 45%|████▍     | 1338/2984 [3:51:16<4:37:06, 10.10s/it]
 45%|████▍     | 1339/2984 [3:51:27<4:36:37, 10.09s/it]
 45%|████▍     | 1340/2984 [3:51:37<4:36:28, 10.09s/it]
                                                       

 45%|████▍     | 1340/2984 [3:51:37<4:36:28, 10.09s/it]
 45%|████▍     | 1341/2984 [3:51:47<4:35:47, 10.07s/it]
 45%|████▍     | 1342/2984 [3:51:57<4:35:03, 10.05s/it]
 45%|████▌     | 1343/2984 [3:52:07<4:34:50, 10.05s/it]
 45%|████▌     | 1344/2984 [3:52:17<4:34:22, 10.04s/it]
 45%|████▌     | 1345/2984 [3:52:27<4:33:29, 10.01s/it]
 45%|████▌     | 1346/2984 [3:52:37<4:3
0: {'loss': 0.5288, 'grad_norm': 0.9852167170639587, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.27}
0: 3:34, 10.02s/it]
 45%|████▌     | 1347/2984 [3:52:47<4:33:55, 10.04s/it]
 45%|████▌     | 1348/2984 [3:52:57<4:33:47, 10.04s/it]
 45%|████▌     | 1349/2984 [3:53:07<4:33:41, 10.04s/it]
 45%|████▌     | 1350/2984 [3:53:17<4:33:33, 10.05s/it]
                                                       

 45%|████▌     | 1350/2984 [3:53:17<4:33:33, 10.05s/it]
 45%|████▌     | 1351/2984 [3:53:27<4:33:44, 10.06s/it]
 45%|████▌     | 1352/2984 [3:53:37<4:33:37, 10.06s/it]
 45%|████▌     | 1353/2984 [3:53:47<4:34:16, 10.09s/it]
 45%|████▌     | 1354/2984 [3:53:57<4:34:16, 10.10s/it]
 45%|████▌     | 1355/2984 [3:54:07<4:33:59, 10.09s/it]
 45%|████▌     | 1356/2984 [3:54:18<4:33:54, 10.09s/it]
 45%|████▌     | 1357/2984 [3:54:28<4:33:17, 10.08s/it]
 46%|████▌     | 1358/2984 [3:54:38<4:33:14, 10.08s/it]
 46%|████▌     | 1359/2984 [3:54:48<4:33:30, 10.10s/it]
 46%|████▌     |
0: {'loss': 0.5378, 'grad_norm': 0.9384784580126897, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.27}
0: {'loss': 0.5332, 'grad_norm': 0.9904233989837526, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.28}
0:  1360/2984 [3:54:58<4:33:57, 10.12s/it]
                                                       

 46%|████▌     | 1360/2984 [3:54:58<4:33:57, 10.12s/it]
 46%|████▌     | 1361/2984 [3:55:08<4:34:13, 10.14s/it]
 46%|████▌     | 1362/2984 [3:55:18<4:34:19, 10.15s/it]
 46%|████▌     | 1363/2984 [3:55:28<4:33:35, 10.13s/it]
 46%|████▌     | 1364/2984 [3:55:38<4:33:18, 10.12s/it]
 46%|████▌     | 1365/2984 [3:55:49<4:33:32, 10.14s/it]
 46%|████▌     | 1366/2984 [3:55:59<4:33:32, 10.14s/it]
 46%|████▌     | 1367/2984 [3:56:09<4:33:31, 10.15s/it]
 46%|████▌     | 1368/2984 [3:56:19<4:33:40, 10.16s/it]
 46%|████▌     | 1369/2984 [3:56:29<4:33:48, 10.17s/it]
 46%|████▌     | 1370/2984 [3:56:40<4:34:41, 10.21s/it]
                                                       

 46%|████▌     | 1370/2984 [3:56:40<4:34:41, 10.21s/it]
 46%|████▌     | 1371/2984 [3:56:50<4:36:04, 10.27s/it]
 46%|██�
0: {'loss': 0.5341, 'grad_norm': 1.0262881536706219, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.28}
0: ��█▌     | 1372/2984 [3:57:01<4:38:06, 10.35s/it]
 46%|████▌     | 1373/2984 [3:57:11<4:39:58, 10.43s/it]
 46%|████▌     | 1374/2984 [3:57:22<4:40:52, 10.47s/it]
 46%|████▌     | 1375/2984 [3:57:32<4:41:37, 10.50s/it]
 46%|████▌     | 1376/2984 [3:57:43<4:42:15, 10.53s/it]
 46%|████▌     | 1377/2984 [3:57:54<4:42:26, 10.55s/it]
 46%|████▌     | 1378/2984 [3:58:04<4:43:03, 10.57s/it]
 46%|████▌     | 1379/2984 [3:58:15<4:42:46, 10.57s/it]
 46%|████▌     | 1380/2984 [3:58:25<4:42:21, 10.56s/it]
                                                       

 46%|████▌     | 1380/2984 [3:58:25<4:42:21, 10.56s/it]
 46%|████▋     | 1381/2984 [3:58:36<4:41:42, 10.54s/it]
 46%|████▋     | 1382/2984 [3:58:46<4:41:14, 10.53s/it]
 46%|████▋     | 1383/2984 [3:58:57<4:41:30, 10.55s/it]
 46%|████▋     | 1384/2984 [3:59:07<4:40:31, 10.52s/it]
 46%|████▋     | 1385/2984 [3:59:18<4:39:29, 
0: {'loss': 0.5248, 'grad_norm': 1.389278084017236, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.28}
0: 10.49s/it]
 46%|████▋     | 1386/2984 [3:59:28<4:39:24, 10.49s/it]
 46%|████▋     | 1387/2984 [3:59:39<4:38:55, 10.48s/it]
 47%|████▋     | 1388/2984 [3:59:49<4:38:47, 10.48s/it]
 47%|████▋     | 1389/2984 [4:00:00<4:38:46, 10.49s/it]
 47%|████▋     | 1390/2984 [4:00:10<4:38:26, 10.48s/it]
                                                       

 47%|████▋     | 1390/2984 [4:00:10<4:38:26, 10.48s/it]
 47%|████▋     | 1391/2984 [4:00:21<4:38:16, 10.48s/it]
 47%|████▋     | 1392/2984 [4:00:31<4:37:00, 10.44s/it]
 47%|████▋     | 1393/2984 [4:00:41<4:35:22, 10.39s/it]
 47%|████▋     | 1394/2984 [4:00:51<4:33:39, 10.33s/it]
 47%|████▋     | 1395/2984 [4:01:02<4:33:07, 10.31s/it]
 47%|████▋     | 1396/2984 [4:01:12<4:32:07, 10.28s/it]
 47%|████▋     | 1397/2984 [4:01:22<4:31:22, 10.26s/it]
 47%|████▋     | 1398/2984 [4:01:32<4:31:17, 10.26s/it]
 47%|████▋     | 1399/
0: {'loss': 0.5293, 'grad_norm': 1.0076489182290274, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.28}
0: {'loss': 0.5142, 'grad_norm': 0.8589270221613554, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.28}
0: 2984 [4:01:43<4:31:39, 10.28s/it]
 47%|████▋     | 1400/2984 [4:01:53<4:31:46, 10.29s/it]
                                                       

 47%|████▋     | 1400/2984 [4:01:53<4:31:46, 10.29s/it]
 47%|████▋     | 1401/2984 [4:02:03<4:31:41, 10.30s/it]
 47%|████▋     | 1402/2984 [4:02:14<4:31:23, 10.29s/it]
 47%|████▋     | 1403/2984 [4:02:24<4:30:50, 10.28s/it]
 47%|████▋     | 1404/2984 [4:02:34<4:30:23, 10.27s/it]
 47%|████▋     | 1405/2984 [4:02:44<4:30:16, 10.27s/it]
 47%|████▋     | 1406/2984 [4:02:55<4:30:00, 10.27s/it]
 47%|████▋     | 1407/2984 [4:03:05<4:30:02, 10.27s/it]
 47%|████▋     | 1408/2984 [4:03:15<4:30:07, 10.28s/it]
 47%|████▋     | 1409/2984 [4:03:26<4:30:54, 10.32s/it]
 47%|████▋     | 1410/2984 [4:03:36<4:33:04, 10.41s/it]
                                                       

 47%|████▋     | 1410/2984 [4:03:36<4:33:04, 10.41s/it]
 47%|████�
0: {'loss': 0.509, 'grad_norm': 1.0569287255386284, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.29}
0: ��     | 1411/2984 [4:03:47<4:31:55, 10.37s/it]
 47%|████▋     | 1412/2984 [4:03:57<4:30:43, 10.33s/it]
 47%|████▋     | 1413/2984 [4:04:07<4:29:41, 10.30s/it]
 47%|████▋     | 1414/2984 [4:04:17<4:28:41, 10.27s/it]
 47%|████▋     | 1415/2984 [4:04:28<4:28:40, 10.27s/it]
 47%|████▋     | 1416/2984 [4:04:38<4:29:10, 10.30s/it]
 47%|████▋     | 1417/2984 [4:04:48<4:29:55, 10.34s/it]
 48%|████▊     | 1418/2984 [4:04:59<4:30:31, 10.36s/it]
 48%|████▊     | 1419/2984 [4:05:09<4:30:29, 10.37s/it]
 48%|████▊     | 1420/2984 [4:05:19<4:30:08, 10.36s/it]
                                                       

 48%|████▊     | 1420/2984 [4:05:19<4:30:08, 10.36s/it]
 48%|████▊     | 1421/2984 [4:05:30<4:29:57, 10.36s/it]
 48%|████▊     | 1422/2984 [4:05:40<4:29:47, 10.36s/it]
 48%|████▊     | 1423/2984 [4:05:51<4:29:33, 10.36s/it]
 48%|████▊     | 1424/2984 [4:06:01<4:30:02, 10.39s
0: {'loss': 0.5197, 'grad_norm': 0.8609341855318683, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.29}
0: /it]
 48%|████▊     | 1425/2984 [4:06:11<4:29:28, 10.37s/it]
 48%|████▊     | 1426/2984 [4:06:22<4:28:33, 10.34s/it]
 48%|████▊     | 1427/2984 [4:06:32<4:27:59, 10.33s/it]
 48%|████▊     | 1428/2984 [4:06:42<4:26:59, 10.30s/it]
 48%|████▊     | 1429/2984 [4:06:52<4:25:21, 10.24s/it]
 48%|████▊     | 1430/2984 [4:07:02<4:24:09, 10.20s/it]
                                                       

 48%|████▊     | 1430/2984 [4:07:02<4:24:09, 10.20s/it]
 48%|████▊     | 1431/2984 [4:07:13<4:24:16, 10.21s/it]
 48%|████▊     | 1432/2984 [4:07:23<4:24:47, 10.24s/it]
 48%|████▊     | 1433/2984 [4:07:33<4:24:52, 10.25s/it]
 48%|████▊     | 1434/2984 [4:07:44<4:29:32, 10.43s/it]
 48%|████▊     | 1435/2984 [4:07:54<4:28:58, 10.42s/it]
 48%|████▊     | 1436/2984 [4:08:05<4:29:09, 10.43s/it]
 48%|████▊     | 1437/2984 [4:08:15<4:28:12, 10.40s/it]
 48%|████▊     | 1438/2984 [
0: {'loss': 0.5281, 'grad_norm': 0.9849481851702466, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.29}
0: {'loss': 0.5247, 'grad_norm': 1.0852321834937495, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.29}
0: 4:08:26<4:28:38, 10.43s/it]
 48%|████▊     | 1439/2984 [4:08:36<4:28:14, 10.42s/it]
 48%|████▊     | 1440/2984 [4:08:47<4:28:22, 10.43s/it]
                                                       

 48%|████▊     | 1440/2984 [4:08:47<4:28:22, 10.43s/it]
 48%|████▊     | 1441/2984 [4:08:57<4:29:02, 10.46s/it]
 48%|████▊     | 1442/2984 [4:09:08<4:29:23, 10.48s/it]
 48%|████▊     | 1443/2984 [4:09:18<4:29:09, 10.48s/it]
 48%|████▊     | 1444/2984 [4:09:29<4:29:22, 10.50s/it]
 48%|████▊     | 1445/2984 [4:09:39<4:29:22, 10.50s/it]
 48%|████▊     | 1446/2984 [4:09:50<4:29:24, 10.51s/it]
 48%|████▊     | 1447/2984 [4:10:00<4:29:32, 10.52s/it]
 49%|████▊     | 1448/2984 [4:10:11<4:28:04, 10.47s/it]
 49%|████▊     | 1449/2984 [4:10:21<4:26:33, 10.42s/it]
 49%|████▊     | 1450/2984 [4:10:31<4:23:53, 10.32s/it]
                                                       

 49%|████▊    
0: {'loss': 0.5135, 'grad_norm': 0.9207808081981206, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.29}
0:  | 1450/2984 [4:10:31<4:23:53, 10.32s/it]
 49%|████▊     | 1451/2984 [4:10:41<4:21:43, 10.24s/it]
 49%|████▊     | 1452/2984 [4:10:51<4:20:08, 10.19s/it]
 49%|████▊     | 1453/2984 [4:11:01<4:19:18, 10.16s/it]
 49%|████▊     | 1454/2984 [4:11:11<4:18:38, 10.14s/it]
 49%|████▉     | 1455/2984 [4:11:21<4:18:12, 10.13s/it]
 49%|████▉     | 1456/2984 [4:11:31<4:18:02, 10.13s/it]
 49%|████▉     | 1457/2984 [4:11:42<4:18:04, 10.14s/it]
 49%|████▉     | 1458/2984 [4:11:52<4:20:36, 10.25s/it]
 49%|████▉     | 1459/2984 [4:12:02<4:19:30, 10.21s/it]
 49%|████▉     | 1460/2984 [4:12:12<4:18:58, 10.20s/it]
                                                       

 49%|████▉     | 1460/2984 [4:12:12<4:18:58, 10.20s/it]
 49%|████▉     | 1461/2984 [4:12:23<4:17:58, 10.16s/it]
 49%|████▉     | 1462/2984 [4:12:33<4:18:05, 10.17s/it]
 49%|████▉     | 1463/2984 [4:12:43<4:18:01, 10.18s/it]
 
0: {'loss': 0.5311, 'grad_norm': 0.9949035615756336, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.3}
0: 49%|████▉     | 1464/2984 [4:12:53<4:18:09, 10.19s/it]
 49%|████▉     | 1465/2984 [4:13:03<4:18:56, 10.23s/it]
 49%|████▉     | 1466/2984 [4:13:14<4:19:16, 10.25s/it]
 49%|████▉     | 1467/2984 [4:13:24<4:18:28, 10.22s/it]
 49%|████▉     | 1468/2984 [4:13:34<4:17:37, 10.20s/it]
 49%|████▉     | 1469/2984 [4:13:44<4:17:04, 10.18s/it]
 49%|████▉     | 1470/2984 [4:13:54<4:17:01, 10.19s/it]
                                                       

 49%|████▉     | 1470/2984 [4:13:54<4:17:01, 10.19s/it]
 49%|████▉     | 1471/2984 [4:14:05<4:16:43, 10.18s/it]
 49%|████▉     | 1472/2984 [4:14:15<4:16:01, 10.16s/it]
 49%|████▉     | 1473/2984 [4:14:25<4:16:33, 10.19s/it]
 49%|████▉     | 1474/2984 [4:14:35<4:17:12, 10.22s/it]
 49%|████▉     | 1475/2984 [4:14:46<4:17:42, 10.25s/it]
 49%|████▉     | 1476/2984 [4:14:56<4:17:06, 10.23s/it]
 49%|████▉     | 1477/2984 [4:15:0
0: {'loss': 0.5342, 'grad_norm': 0.9952339707254982, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.3}
0: 6<4:15:44, 10.18s/it]
 50%|████▉     | 1478/2984 [4:15:16<4:15:18, 10.17s/it]
 50%|████▉     | 1479/2984 [4:15:26<4:15:18, 10.18s/it]
 50%|████▉     | 1480/2984 [4:15:36<4:15:04, 10.18s/it]
                                                       

 50%|████▉     | 1480/2984 [4:15:36<4:15:04, 10.18s/it]
 50%|████▉     | 1481/2984 [4:15:46<4:14:53, 10.18s/it]
 50%|████▉     | 1482/2984 [4:15:57<4:14:15, 10.16s/it]
 50%|████▉     | 1483/2984 [4:16:07<4:14:15, 10.16s/it]
 50%|████▉     | 1484/2984 [4:16:17<4:15:14, 10.21s/it]
 50%|████▉     | 1485/2984 [4:16:27<4:15:43, 10.24s/it]
 50%|████▉     | 1486/2984 [4:16:38<4:16:02, 10.26s/it]
 50%|████▉     | 1487/2984 [4:16:48<4:16:18, 10.27s/it]
 50%|████▉     | 1488/2984 [4:16:58<4:16:44, 10.30s/it]
 50%|████▉     | 1489/2984 [4:17:09<4:17:31, 10.34s/it]
 50%|████▉     | 1490/2984 [4:17:19<4:18:17, 10.37s/it]
                     
0: {'loss': 0.528, 'grad_norm': 0.8706294723812696, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.3}
0: {'loss': 0.5176, 'grad_norm': 0.898669693298431, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.3}
0:                                   

 50%|████▉     | 1490/2984 [4:17:19<4:18:17, 10.37s/it]
 50%|████▉     | 1491/2984 [4:17:30<4:18:48, 10.40s/it]
 50%|█████     | 1492/2984 [4:17:40<4:18:46, 10.41s/it]
 50%|█████     | 1493/2984 [4:17:51<4:18:48, 10.41s/it]
 50%|█████     | 1494/2984 [4:18:01<4:18:57, 10.43s/it]
 50%|█████     | 1495/2984 [4:18:12<4:19:28, 10.46s/it]
 50%|█████     | 1496/2984 [4:18:22<4:19:44, 10.47s/it]
 50%|█████     | 1497/2984 [4:18:33<4:19:54, 10.49s/it]
 50%|█████     | 1498/2984 [4:18:43<4:20:00, 10.50s/it]
 50%|█████     | 1499/2984 [4:18:54<4:19:36, 10.49s/it]
 50%|█████     | 1500/2984 [4:19:04<4:19:22, 10.49s/it]
                                                       

 50%|█████     | 1500/2984 [4:19:04<4:19:22, 10.49s/it]
 50%|█████     | 1501/2984 [4:19:14<4:18:21, 10.45s/it]
 50%|█████     | 1502/2984 [4:19:25<4:17:01, 10.41s/it]
 50%|�
0: {'loss': 0.511, 'grad_norm': 0.9294095709918129, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.3}
0: �████     | 1503/2984 [4:19:35<4:16:28, 10.39s/it]
 50%|█████     | 1504/2984 [4:19:45<4:16:16, 10.39s/it]
 50%|█████     | 1505/2984 [4:19:56<4:16:11, 10.39s/it]
 50%|█████     | 1506/2984 [4:20:06<4:14:55, 10.35s/it]
 51%|█████     | 1507/2984 [4:20:16<4:13:38, 10.30s/it]
 51%|█████     | 1508/2984 [4:20:26<4:12:10, 10.25s/it]
 51%|█████     | 1509/2984 [4:20:36<4:10:36, 10.19s/it]
 51%|█████     | 1510/2984 [4:20:47<4:10:52, 10.21s/it]
                                                       

 51%|█████     | 1510/2984 [4:20:47<4:10:52, 10.21s/it]
 51%|█████     | 1511/2984 [4:20:57<4:11:28, 10.24s/it]
 51%|█████     | 1512/2984 [4:21:08<4:13:15, 10.32s/it]
 51%|█████     | 1513/2984 [4:21:18<4:14:05, 10.36s/it]
 51%|█████     | 1514/2984 [4:21:29<4:15:09, 10.41s/it]
 51%|█████     | 1515/2984 [4:21:39<4:15:19, 10.43s/it]
 51%|█████     | 1516/2984 [4:21:50<4:17
0: {'loss': 0.5148, 'grad_norm': 0.8875680302486909, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.31}
0: :00, 10.50s/it]
 51%|█████     | 1517/2984 [4:22:00<4:16:34, 10.49s/it]
 51%|█████     | 1518/2984 [4:22:11<4:16:42, 10.51s/it]
 51%|█████     | 1519/2984 [4:22:21<4:16:57, 10.52s/it]
 51%|█████     | 1520/2984 [4:22:32<4:16:46, 10.52s/it]
                                                       

 51%|█████     | 1520/2984 [4:22:32<4:16:46, 10.52s/it]
 51%|█████     | 1521/2984 [4:22:42<4:14:20, 10.43s/it]
 51%|█████     | 1522/2984 [4:22:52<4:12:41, 10.37s/it]
 51%|█████     | 1523/2984 [4:23:02<4:10:51, 10.30s/it]
 51%|█████     | 1524/2984 [4:23:12<4:09:17, 10.24s/it]
 51%|█████     | 1525/2984 [4:23:23<4:08:32, 10.22s/it]
 51%|█████     | 1526/2984 [4:23:33<4:07:09, 10.17s/it]
 51%|█████     | 1527/2984 [4:23:43<4:06:58, 10.17s/it]
 51%|█████     | 1528/2984 [4:23:53<4:06:42, 10.17s/it]
 51%|█████     | 1529/2984 [4:24:03<4:07:01, 10.19s/it]
 51%|█████▏    
0: {'loss': 0.5072, 'grad_norm': 0.8781112945257018, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.31}
0: {'loss': 0.5131, 'grad_norm': 1.080553253409831, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.31}
0: | 1530/2984 [4:24:13<4:07:09, 10.20s/it]
                                                       

 51%|█████▏    | 1530/2984 [4:24:13<4:07:09, 10.20s/it]
 51%|█████▏    | 1531/2984 [4:24:24<4:06:57, 10.20s/it]
 51%|█████▏    | 1532/2984 [4:24:34<4:07:28, 10.23s/it]
 51%|█████▏    | 1533/2984 [4:24:44<4:07:20, 10.23s/it]
 51%|█████▏    | 1534/2984 [4:24:54<4:06:51, 10.21s/it]
 51%|█████▏    | 1535/2984 [4:25:05<4:06:48, 10.22s/it]
 51%|█████▏    | 1536/2984 [4:25:15<4:06:55, 10.23s/it]
 52%|█████▏    | 1537/2984 [4:25:25<4:06:44, 10.23s/it]
 52%|█████▏    | 1538/2984 [4:25:35<4:06:39, 10.23s/it]
 52%|█████▏    | 1539/2984 [4:25:46<4:06:16, 10.23s/it]
 52%|█████▏    | 1540/2984 [4:25:56<4:05:26, 10.20s/it]
                                                       

 52%|█████▏    | 1540/2984 [4:25:56<4:05:26, 10.20s/it]
 52%|█████▏    | 1541/2984 [4:26:06<4:05:
0: {'loss': 0.5262, 'grad_norm': 0.8983748932765903, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.31}
0: 11, 10.20s/it]
 52%|█████▏    | 1542/2984 [4:26:17<4:08:28, 10.34s/it]
 52%|█████▏    | 1543/2984 [4:26:27<4:09:49, 10.40s/it]
 52%|█████▏    | 1544/2984 [4:26:37<4:08:26, 10.35s/it]
 52%|█████▏    | 1545/2984 [4:26:47<4:06:39, 10.28s/it]
 52%|█████▏    | 1546/2984 [4:26:58<4:05:03, 10.22s/it]
 52%|█████▏    | 1547/2984 [4:27:07<4:02:53, 10.14s/it]
 52%|█████▏    | 1548/2984 [4:27:18<4:02:02, 10.11s/it]
 52%|█████▏    | 1549/2984 [4:27:28<4:01:36, 10.10s/it]
 52%|█████▏    | 1550/2984 [4:27:38<4:01:38, 10.11s/it]
                                                       

 52%|█████▏    | 1550/2984 [4:27:38<4:01:38, 10.11s/it]
 52%|█████▏    | 1551/2984 [4:27:48<4:01:53, 10.13s/it]
 52%|█████▏    | 1552/2984 [4:27:58<4:01:31, 10.12s/it]
 52%|█████▏    | 1553/2984 [4:28:08<4:00:17, 10.08s/it]
 52%|█████▏    | 1554/2984 [4:28:18<3:59:21, 10.04s/it]
0: {'loss': 0.5165, 'grad_norm': 1.0219851060979561, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.31}
0:  52%|█████▏    | 1555/2984 [4:28:28<3:58:27, 10.01s/it]
 52%|█████▏    | 1556/2984 [4:28:38<3:58:02, 10.00s/it]
 52%|█████▏    | 1557/2984 [4:28:48<3:57:35,  9.99s/it]
 52%|█████▏    | 1558/2984 [4:28:58<3:57:48, 10.01s/it]
 52%|█████▏    | 1559/2984 [4:29:08<3:58:13, 10.03s/it]
 52%|█████▏    | 1560/2984 [4:29:18<3:58:29, 10.05s/it]
                                                       

 52%|█████▏    | 1560/2984 [4:29:18<3:58:29, 10.05s/it]
 52%|█████▏    | 1561/2984 [4:29:28<3:58:28, 10.05s/it]
 52%|█████▏    | 1562/2984 [4:29:38<3:58:41, 10.07s/it]
 52%|█████▏    | 1563/2984 [4:29:48<3:58:44, 10.08s/it]
 52%|█████▏    | 1564/2984 [4:29:58<3:58:31, 10.08s/it]
 52%|█████▏    | 1565/2984 [4:30:09<3:58:31, 10.09s/it]
 52%|█████▏    | 1566/2984 [4:30:19<3:59:02, 10.11s/it]
 53%|█████▎    | 1567/2984 [4:30:29<3:59:40, 10.15s/it]
 53%|███�
0: {'loss': 0.5126, 'grad_norm': 0.9007565880788946, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.32}
0: ��█▎    | 1568/2984 [4:30:39<3:58:52, 10.12s/it]
 53%|█████▎    | 1569/2984 [4:30:49<3:58:30, 10.11s/it]
 53%|█████▎    | 1570/2984 [4:30:59<3:58:04, 10.10s/it]
                                                       

 53%|█████▎    | 1570/2984 [4:30:59<3:58:04, 10.10s/it]
 53%|█████▎    | 1571/2984 [4:31:09<3:57:58, 10.11s/it]
 53%|█████▎    | 1572/2984 [4:31:19<3:57:18, 10.08s/it]
 53%|█████▎    | 1573/2984 [4:31:29<3:56:56, 10.08s/it]
 53%|█████▎    | 1574/2984 [4:31:39<3:56:51, 10.08s/it]
 53%|█████▎    | 1575/2984 [4:31:50<3:56:40, 10.08s/it]
 53%|█████▎    | 1576/2984 [4:32:00<3:56:16, 10.07s/it]
 53%|█████▎    | 1577/2984 [4:32:09<3:55:07, 10.03s/it]
 53%|█████▎    | 1578/2984 [4:32:19<3:54:01,  9.99s/it]
 53%|█████▎    | 1579/2984 [4:32:29<3:53:57,  9.99s/it]
 53%|█████▎    | 1580/2984 [4:32:39<3:53:11,  9.97s/it]
                              
0: {'loss': 0.508, 'grad_norm': 0.9336708474698726, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.32}
0: {'loss': 0.5243, 'grad_norm': 0.9857807540375876, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.32}
0:                          

 53%|█████▎    | 1580/2984 [4:32:39<3:53:11,  9.97s/it]
 53%|█████▎    | 1581/2984 [4:32:49<3:52:54,  9.96s/it]
 53%|█████▎    | 1582/2984 [4:32:59<3:52:51,  9.97s/it]
 53%|█████▎    | 1583/2984 [4:33:09<3:53:20,  9.99s/it]
 53%|█████▎    | 1584/2984 [4:33:19<3:53:36, 10.01s/it]
 53%|█████▎    | 1585/2984 [4:33:29<3:53:29, 10.01s/it]
 53%|█████▎    | 1586/2984 [4:33:39<3:53:16, 10.01s/it]
 53%|█████▎    | 1587/2984 [4:33:49<3:53:37, 10.03s/it]
 53%|█████▎    | 1588/2984 [4:33:59<3:53:03, 10.02s/it]
 53%|█████▎    | 1589/2984 [4:34:10<3:53:21, 10.04s/it]
 53%|█████▎    | 1590/2984 [4:34:20<3:53:24, 10.05s/it]
                                                       

 53%|█████▎    | 1590/2984 [4:34:20<3:53:24, 10.05s/it]
 53%|█████▎    | 1591/2984 [4:34:30<3:53:07, 10.04s/it]
 53%|█████▎    | 1592/2984 [4:34:40<3:54:32,
0: {'loss': 0.5144, 'grad_norm': 0.9212945575858148, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.32}
0:  10.11s/it]
 53%|█████▎    | 1593/2984 [4:34:50<3:54:58, 10.14s/it]
 53%|█████▎    | 1594/2984 [4:35:00<3:55:52, 10.18s/it]
 53%|█████▎    | 1595/2984 [4:35:11<3:55:53, 10.19s/it]
 53%|█████▎    | 1596/2984 [4:35:21<3:55:48, 10.19s/it]
 54%|█████▎    | 1597/2984 [4:35:31<3:55:45, 10.20s/it]
 54%|█████▎    | 1598/2984 [4:35:41<3:56:08, 10.22s/it]
 54%|█████▎    | 1599/2984 [4:35:52<3:56:59, 10.27s/it]
 54%|█████▎    | 1600/2984 [4:36:02<3:56:18, 10.24s/it]
                                                       

 54%|█████▎    | 1600/2984 [4:36:02<3:56:18, 10.24s/it]
 54%|█████▎    | 1601/2984 [4:36:12<3:55:53, 10.23s/it]
 54%|█████▎    | 1602/2984 [4:36:22<3:56:17, 10.26s/it]
 54%|█████▎    | 1603/2984 [4:36:33<3:56:52, 10.29s/it]
 54%|█████▍    | 1604/2984 [4:36:43<3:57:12, 10.31s/it]
 54%|█████▍    | 1605/2984 [4:36:53<3:56:25, 10.29s/it]
 54
0: {'loss': 0.5121, 'grad_norm': 0.9848451678222024, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.32}
0: %|█████▍    | 1606/2984 [4:37:04<3:56:36, 10.30s/it]
 54%|█████▍    | 1607/2984 [4:37:14<3:56:33, 10.31s/it]
 54%|█████▍    | 1608/2984 [4:37:24<3:53:51, 10.20s/it]
 54%|█████▍    | 1609/2984 [4:37:34<3:52:02, 10.13s/it]
 54%|█████▍    | 1610/2984 [4:37:44<3:50:56, 10.09s/it]
                                                       

 54%|█████▍    | 1610/2984 [4:37:44<3:50:56, 10.09s/it]
 54%|█████▍    | 1611/2984 [4:37:54<3:50:54, 10.09s/it]
 54%|█████▍    | 1612/2984 [4:38:04<3:50:15, 10.07s/it]
 54%|█████▍    | 1613/2984 [4:38:14<3:49:10, 10.03s/it]
 54%|█████▍    | 1614/2984 [4:38:24<3:49:29, 10.05s/it]
 54%|█████▍    | 1615/2984 [4:38:34<3:49:57, 10.08s/it]
 54%|█████▍    | 1616/2984 [4:38:44<3:49:33, 10.07s/it]
 54%|█████▍    | 1617/2984 [4:38:54<3:49:44, 10.08s/it]
 54%|█████▍    | 1618/2984 [4:39:04<3:49:40, 10.09s/it]
 54%|████�
0: {'loss': 0.5309, 'grad_norm': 0.88361445124227, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.33}
0: {'loss': 0.5104, 'grad_norm': 1.0454890869055882, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.33}
0: ��▍    | 1619/2984 [4:39:15<3:49:32, 10.09s/it]
 54%|█████▍    | 1620/2984 [4:39:25<3:48:47, 10.06s/it]
                                                       

 54%|█████▍    | 1620/2984 [4:39:25<3:48:47, 10.06s/it]
 54%|█████▍    | 1621/2984 [4:39:35<3:48:30, 10.06s/it]
 54%|█████▍    | 1622/2984 [4:39:45<3:48:30, 10.07s/it]
 54%|█████▍    | 1623/2984 [4:39:55<3:48:41, 10.08s/it]
 54%|█████▍    | 1624/2984 [4:40:05<3:51:40, 10.22s/it]
 54%|█████▍    | 1625/2984 [4:40:16<3:54:50, 10.37s/it]
 54%|█████▍    | 1626/2984 [4:40:26<3:54:33, 10.36s/it]
 55%|█████▍    | 1627/2984 [4:40:36<3:52:43, 10.29s/it]
 55%|█████▍    | 1628/2984 [4:40:47<3:52:54, 10.31s/it]
 55%|█████▍    | 1629/2984 [4:40:57<3:53:23, 10.33s/it]
 55%|█████▍    | 1630/2984 [4:41:08<3:54:13, 10.38s/it]
                                                       

 55%|█████▍    | 1630/2984 [4:41
0: {'loss': 0.5321, 'grad_norm': 0.9267211416299209, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.33}
0: :08<3:54:13, 10.38s/it]
 55%|█████▍    | 1631/2984 [4:41:18<3:54:37, 10.40s/it]
 55%|█████▍    | 1632/2984 [4:41:29<3:55:51, 10.47s/it]
 55%|█████▍    | 1633/2984 [4:41:39<3:55:05, 10.44s/it]
 55%|█████▍    | 1634/2984 [4:41:49<3:53:41, 10.39s/it]
 55%|█████▍    | 1635/2984 [4:42:00<3:52:58, 10.36s/it]
 55%|█████▍    | 1636/2984 [4:42:10<3:52:51, 10.36s/it]
 55%|█████▍    | 1637/2984 [4:42:20<3:52:23, 10.35s/it]
 55%|█████▍    | 1638/2984 [4:42:31<3:52:11, 10.35s/it]
 55%|█████▍    | 1639/2984 [4:42:41<3:53:00, 10.39s/it]
 55%|█████▍    | 1640/2984 [4:42:52<3:52:40, 10.39s/it]
                                                       

 55%|█████▍    | 1640/2984 [4:42:52<3:52:40, 10.39s/it]
 55%|█████▍    | 1641/2984 [4:43:02<3:52:09, 10.37s/it]
 55%|█████▌    | 1642/2984 [4:43:12<3:51:46, 10.36s/it]
 55%|█████▌    | 1643/2984 [4:43:23<3:51:41, 10
0: {'loss': 0.523, 'grad_norm': 0.9376822701411395, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.33}
0: .37s/it]
 55%|█████▌    | 1644/2984 [4:43:33<3:51:27, 10.36s/it]
 55%|█████▌    | 1645/2984 [4:43:43<3:50:26, 10.33s/it]
 55%|█████▌    | 1646/2984 [4:43:54<3:49:48, 10.31s/it]
 55%|█████▌    | 1647/2984 [4:44:04<3:49:55, 10.32s/it]
 55%|█████▌    | 1648/2984 [4:44:14<3:49:55, 10.33s/it]
 55%|█████▌    | 1649/2984 [4:44:25<3:50:03, 10.34s/it]
 55%|█████▌    | 1650/2984 [4:44:35<3:49:23, 10.32s/it]
                                                       

 55%|█████▌    | 1650/2984 [4:44:35<3:49:23, 10.32s/it]
 55%|█████▌    | 1651/2984 [4:44:45<3:49:26, 10.33s/it]
 55%|█████▌    | 1652/2984 [4:44:56<3:49:21, 10.33s/it]
 55%|█████▌    | 1653/2984 [4:45:06<3:48:16, 10.29s/it]
 55%|█████▌    | 1654/2984 [4:45:16<3:48:01, 10.29s/it]
 55%|█████▌    | 1655/2984 [4:45:26<3:47:25, 10.27s/it]
 55%|█████▌    | 1656/2984 [4:45:36<3:47:00, 10.26s/it]
 56%|�
0: {'loss': 0.5215, 'grad_norm': 0.9395566022207101, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.33}
0: ��████▌    | 1657/2984 [4:45:47<3:45:28, 10.19s/it]
 56%|█████▌    | 1658/2984 [4:45:57<3:44:31, 10.16s/it]
 56%|█████▌    | 1659/2984 [4:46:07<3:44:28, 10.17s/it]
 56%|█████▌    | 1660/2984 [4:46:17<3:45:11, 10.21s/it]
                                                       

 56%|█████▌    | 1660/2984 [4:46:17<3:45:11, 10.21s/it]
 56%|█████▌    | 1661/2984 [4:46:28<3:46:39, 10.28s/it]
 56%|█████▌    | 1662/2984 [4:46:38<3:46:51, 10.30s/it]
 56%|█████▌    | 1663/2984 [4:46:48<3:47:36, 10.34s/it]
 56%|█████▌    | 1664/2984 [4:46:59<3:48:37, 10.39s/it]
 56%|█████▌    | 1665/2984 [4:47:09<3:49:46, 10.45s/it]
 56%|█████▌    | 1666/2984 [4:47:20<3:51:18, 10.53s/it]
 56%|█████▌    | 1667/2984 [4:47:31<3:52:06, 10.57s/it]
 56%|█████▌    | 1668/2984 [4:47:42<3:53:17, 10.64s/it]
 56%|█████▌    | 1669/2984 [4:47:52<3:52:41, 10.62s/it]
 56%|█████�
0: {'loss': 0.5202, 'grad_norm': 1.0210204441342159, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.34}
0: {'loss': 0.5171, 'grad_norm': 0.9476105494354089, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.34}
0: ��    | 1670/2984 [4:48:03<3:51:34, 10.57s/it]
                                                       

 56%|█████▌    | 1670/2984 [4:48:03<3:51:34, 10.57s/it]
 56%|█████▌    | 1671/2984 [4:48:13<3:50:24, 10.53s/it]
 56%|█████▌    | 1672/2984 [4:48:23<3:49:13, 10.48s/it]
 56%|█████▌    | 1673/2984 [4:48:34<3:49:25, 10.50s/it]
 56%|█████▌    | 1674/2984 [4:48:44<3:47:48, 10.43s/it]
 56%|█████▌    | 1675/2984 [4:48:55<3:46:46, 10.39s/it]
 56%|█████▌    | 1676/2984 [4:49:05<3:45:35, 10.35s/it]
 56%|█████▌    | 1677/2984 [4:49:15<3:44:42, 10.32s/it]
 56%|█████▌    | 1678/2984 [4:49:25<3:42:30, 10.22s/it]
 56%|█████▋    | 1679/2984 [4:49:35<3:41:16, 10.17s/it]
 56%|█████▋    | 1680/2984 [4:49:45<3:40:38, 10.15s/it]
                                                       

 56%|█████▋    | 1680/2984 [4:49:45<3:40:38, 10.15s/it]
 56%|█████▋    | 1681/2984 [4:49:55
0: {'loss': 0.5054, 'grad_norm': 1.0047554907512257, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.34}
0: <3:40:16, 10.14s/it]
 56%|█████▋    | 1682/2984 [4:50:05<3:39:16, 10.11s/it]
 56%|█████▋    | 1683/2984 [4:50:16<3:39:20, 10.12s/it]
 56%|█████▋    | 1684/2984 [4:50:26<3:39:44, 10.14s/it]
 56%|█████▋    | 1685/2984 [4:50:36<3:38:37, 10.10s/it]
 57%|█████▋    | 1686/2984 [4:50:46<3:37:13, 10.04s/it]
 57%|█████▋    | 1687/2984 [4:50:56<3:36:35, 10.02s/it]
 57%|█████▋    | 1688/2984 [4:51:06<3:35:59, 10.00s/it]
 57%|█████▋    | 1689/2984 [4:51:16<3:35:48, 10.00s/it]
 57%|█████▋    | 1690/2984 [4:51:26<3:36:48, 10.05s/it]
                                                       

 57%|█████▋    | 1690/2984 [4:51:26<3:36:48, 10.05s/it]
 57%|█████▋    | 1691/2984 [4:51:36<3:36:43, 10.06s/it]
 57%|█████▋    | 1692/2984 [4:51:46<3:36:54, 10.07s/it]
 57%|█████▋    | 1693/2984 [4:51:56<3:36:35, 10.07s/it]
 57%|█████▋    | 1694/2984 [4:52:06<3:36:10, 10.05
0: {'loss': 0.5186, 'grad_norm': 0.9511389102017296, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.34}
0: s/it]
 57%|█████▋    | 1695/2984 [4:52:16<3:35:59, 10.05s/it]
 57%|█████▋    | 1696/2984 [4:52:26<3:35:32, 10.04s/it]
 57%|█████▋    | 1697/2984 [4:52:36<3:35:28, 10.05s/it]
 57%|█████▋    | 1698/2984 [4:52:46<3:35:27, 10.05s/it]
 57%|█████▋    | 1699/2984 [4:52:56<3:35:04, 10.04s/it]
 57%|█████▋    | 1700/2984 [4:53:06<3:34:43, 10.03s/it]
                                                       

 57%|█████▋    | 1700/2984 [4:53:06<3:34:43, 10.03s/it]
 57%|█████▋    | 1701/2984 [4:53:16<3:35:02, 10.06s/it]
 57%|█████▋    | 1702/2984 [4:53:26<3:34:58, 10.06s/it]
 57%|█████▋    | 1703/2984 [4:53:36<3:34:22, 10.04s/it]
 57%|█████▋    | 1704/2984 [4:53:47<3:35:01, 10.08s/it]
 57%|█████▋    | 1705/2984 [4:53:57<3:34:57, 10.08s/it]
 57%|█████▋    | 1706/2984 [4:54:07<3:34:15, 10.06s/it]
 57%|█████▋    | 1707/2984 [4:54:17<3:33:47, 10.05s/it]
 57%|█�
0: {'loss': 0.5152, 'grad_norm': 0.9228161562992192, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.34}
0: ��███▋    | 1708/2984 [4:54:27<3:34:45, 10.10s/it]
 57%|█████▋    | 1709/2984 [4:54:37<3:35:48, 10.16s/it]
 57%|█████▋    | 1710/2984 [4:54:47<3:36:36, 10.20s/it]
                                                       

 57%|█████▋    | 1710/2984 [4:54:47<3:36:36, 10.20s/it]
 57%|█████▋    | 1711/2984 [4:54:58<3:36:51, 10.22s/it]
 57%|█████▋    | 1712/2984 [4:55:08<3:36:56, 10.23s/it]
 57%|█████▋    | 1713/2984 [4:55:19<3:38:42, 10.32s/it]
 57%|█████▋    | 1714/2984 [4:55:29<3:37:47, 10.29s/it]
 57%|█████▋    | 1715/2984 [4:55:39<3:36:42, 10.25s/it]
 58%|█████▊    | 1716/2984 [4:55:49<3:35:59, 10.22s/it]
 58%|█████▊    | 1717/2984 [4:55:59<3:35:29, 10.21s/it]
 58%|█████▊    | 1718/2984 [4:56:09<3:35:11, 10.20s/it]
 58%|█████▊    | 1719/2984 [4:56:20<3:34:54, 10.19s/it]
 58%|█████▊    | 1720/2984 [4:56:30<3:34:27, 10.18s/it]
                        
0: {'loss': 0.5106, 'grad_norm': 0.9583876280611756, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.35}
0: {'loss': 0.5071, 'grad_norm': 0.8664719707909116, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.35}
0:                                

 58%|█████▊    | 1720/2984 [4:56:30<3:34:27, 10.18s/it]
 58%|█████▊    | 1721/2984 [4:56:40<3:34:09, 10.17s/it]
 58%|█████▊    | 1722/2984 [4:56:50<3:33:31, 10.15s/it]
 58%|█████▊    | 1723/2984 [4:57:00<3:33:07, 10.14s/it]
 58%|█████▊    | 1724/2984 [4:57:10<3:32:10, 10.10s/it]
 58%|█████▊    | 1725/2984 [4:57:20<3:31:48, 10.09s/it]
 58%|█████▊    | 1726/2984 [4:57:30<3:31:20, 10.08s/it]
 58%|█████▊    | 1727/2984 [4:57:41<3:35:50, 10.30s/it]
 58%|█████▊    | 1728/2984 [4:57:51<3:34:06, 10.23s/it]
 58%|█████▊    | 1729/2984 [4:58:01<3:33:20, 10.20s/it]
 58%|█████▊    | 1730/2984 [4:58:12<3:35:57, 10.33s/it]
                                                       

 58%|█████▊    | 1730/2984 [4:58:12<3:35:57, 10.33s/it]
 58%|█████▊    | 1731/2984 [4:58:22<3:34:35, 10.28s/it]
 58%|█████▊    | 1732/2984 [4:58:33<3:
0: {'loss': 0.4969, 'grad_norm': 0.9260757777530901, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.35}
0: 41:14, 10.60s/it]
 58%|█████▊    | 1733/2984 [4:58:44<3:39:29, 10.53s/it]
 58%|█████▊    | 1734/2984 [4:58:54<3:39:41, 10.55s/it]
 58%|█████▊    | 1735/2984 [4:59:04<3:36:36, 10.41s/it]
 58%|█████▊    | 1736/2984 [4:59:15<3:35:59, 10.38s/it]
 58%|█████▊    | 1737/2984 [4:59:25<3:33:14, 10.26s/it]
 58%|█████▊    | 1738/2984 [4:59:35<3:31:13, 10.17s/it]
 58%|█████▊    | 1739/2984 [4:59:45<3:29:48, 10.11s/it]
 58%|█████▊    | 1740/2984 [4:59:55<3:29:01, 10.08s/it]
                                                       

 58%|█████▊    | 1740/2984 [4:59:55<3:29:01, 10.08s/it]
 58%|█████▊    | 1741/2984 [5:00:05<3:28:38, 10.07s/it]
 58%|█████▊    | 1742/2984 [5:00:15<3:28:08, 10.06s/it]
 58%|█████▊    | 1743/2984 [5:00:25<3:27:53, 10.05s/it]
 58%|█████▊    | 1744/2984 [5:00:35<3:27:35, 10.04s/it]
 58%|█████▊    | 1745/2984 [5:00:45<3:27:42, 10.06s/i
0: {'loss': 0.5046, 'grad_norm': 0.9486850739820486, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.35}
0: t]
 59%|█████▊    | 1746/2984 [5:00:55<3:28:16, 10.09s/it]
 59%|█████▊    | 1747/2984 [5:01:05<3:29:10, 10.15s/it]
 59%|█████▊    | 1748/2984 [5:01:16<3:29:44, 10.18s/it]
 59%|█████▊    | 1749/2984 [5:01:26<3:30:08, 10.21s/it]
 59%|█████▊    | 1750/2984 [5:01:36<3:30:16, 10.22s/it]
                                                       

 59%|█████▊    | 1750/2984 [5:01:36<3:30:16, 10.22s/it]
 59%|█████▊    | 1751/2984 [5:01:47<3:31:52, 10.31s/it]
 59%|█████▊    | 1752/2984 [5:01:57<3:32:16, 10.34s/it]
 59%|█████▊    | 1753/2984 [5:02:07<3:31:51, 10.33s/it]
 59%|█████▉    | 1754/2984 [5:02:18<3:33:25, 10.41s/it]
 59%|█████▉    | 1755/2984 [5:02:28<3:32:08, 10.36s/it]
 59%|█████▉    | 1756/2984 [5:02:39<3:32:33, 10.39s/it]
 59%|█████▉    | 1757/2984 [5:02:49<3:32:20, 10.38s/it]
 59%|█████▉    | 1758/2984 [5:02:59<3:30:47, 10.32s/it]
 59%|██�
0: {'loss': 0.5181, 'grad_norm': 1.240579504751822, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.35}
0: {'loss': 0.5227, 'grad_norm': 0.8492031851787167, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.36}
0: ��██▉    | 1759/2984 [5:03:09<3:29:54, 10.28s/it]
 59%|█████▉    | 1760/2984 [5:03:20<3:29:02, 10.25s/it]
                                                       

 59%|█████▉    | 1760/2984 [5:03:20<3:29:02, 10.25s/it]
 59%|█████▉    | 1761/2984 [5:03:30<3:28:34, 10.23s/it]
 59%|█████▉    | 1762/2984 [5:03:40<3:28:12, 10.22s/it]
 59%|█████▉    | 1763/2984 [5:03:50<3:27:25, 10.19s/it]
 59%|█████▉    | 1764/2984 [5:04:00<3:26:50, 10.17s/it]
 59%|█████▉    | 1765/2984 [5:04:10<3:26:20, 10.16s/it]
 59%|█████▉    | 1766/2984 [5:04:20<3:25:55, 10.14s/it]
 59%|█████▉    | 1767/2984 [5:04:31<3:25:34, 10.14s/it]
 59%|█████▉    | 1768/2984 [5:04:41<3:26:06, 10.17s/it]
 59%|█████▉    | 1769/2984 [5:04:51<3:25:36, 10.15s/it]
 59%|█████▉    | 1770/2984 [5:05:01<3:25:18, 10.15s/it]
                                                       

 59%|█████▉    | 1770/2984
0: {'loss': 0.5092, 'grad_norm': 0.8835718086213576, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.36}
0:  [5:05:01<3:25:18, 10.15s/it]
 59%|█████▉    | 1771/2984 [5:05:11<3:25:40, 10.17s/it]
 59%|█████▉    | 1772/2984 [5:05:21<3:25:44, 10.19s/it]
 59%|█████▉    | 1773/2984 [5:05:32<3:24:59, 10.16s/it]
 59%|█████▉    | 1774/2984 [5:05:42<3:23:56, 10.11s/it]
 59%|█████▉    | 1775/2984 [5:05:52<3:23:47, 10.11s/it]
 60%|█████▉    | 1776/2984 [5:06:02<3:24:05, 10.14s/it]
 60%|█████▉    | 1777/2984 [5:06:12<3:24:13, 10.15s/it]
 60%|█████▉    | 1778/2984 [5:06:22<3:24:36, 10.18s/it]
 60%|█████▉    | 1779/2984 [5:06:33<3:25:26, 10.23s/it]
 60%|█████▉    | 1780/2984 [5:06:43<3:25:58, 10.26s/it]
                                                       

 60%|█████▉    | 1780/2984 [5:06:43<3:25:58, 10.26s/it]
 60%|█████▉    | 1781/2984 [5:06:53<3:26:58, 10.32s/it]
 60%|█████▉    | 1782/2984 [5:07:04<3:29:02, 10.43s/it]
 60%|█████▉    | 1783/2984 [5:07:15<3:29:
0: {'loss': 0.5022, 'grad_norm': 0.9240716200732269, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.36}
0: 31, 10.47s/it]
 60%|█████▉    | 1784/2984 [5:07:25<3:28:38, 10.43s/it]
 60%|█████▉    | 1785/2984 [5:07:35<3:27:49, 10.40s/it]
 60%|█████▉    | 1786/2984 [5:07:46<3:27:01, 10.37s/it]
 60%|█████▉    | 1787/2984 [5:07:56<3:26:43, 10.36s/it]
 60%|█████▉    | 1788/2984 [5:08:06<3:26:00, 10.33s/it]
 60%|█████▉    | 1789/2984 [5:08:17<3:25:45, 10.33s/it]
 60%|█████▉    | 1790/2984 [5:08:27<3:24:32, 10.28s/it]
                                                       

 60%|█████▉    | 1790/2984 [5:08:27<3:24:32, 10.28s/it]
 60%|██████    | 1791/2984 [5:08:37<3:24:05, 10.26s/it]
 60%|██████    | 1792/2984 [5:08:47<3:23:37, 10.25s/it]
 60%|██████    | 1793/2984 [5:08:57<3:23:03, 10.23s/it]
 60%|██████    | 1794/2984 [5:09:08<3:22:23, 10.20s/it]
 60%|██████    | 1795/2984 [5:09:18<3:22:00, 10.19s/it]
 60%|██████    | 1796/2984 [5:09:28<3:20:33, 10.13s/it]
0: {'loss': 0.5178, 'grad_norm': 0.9548002647217907, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.36}
0:  60%|██████    | 1797/2984 [5:09:38<3:20:02, 10.11s/it]
 60%|██████    | 1798/2984 [5:09:48<3:19:24, 10.09s/it]
 60%|██████    | 1799/2984 [5:09:58<3:19:36, 10.11s/it]
 60%|██████    | 1800/2984 [5:10:08<3:19:52, 10.13s/it]
                                                       

 60%|██████    | 1800/2984 [5:10:08<3:19:52, 10.13s/it]
 60%|██████    | 1801/2984 [5:10:18<3:20:02, 10.15s/it]
 60%|██████    | 1802/2984 [5:10:28<3:19:59, 10.15s/it]
 60%|██████    | 1803/2984 [5:10:39<3:19:37, 10.14s/it]
 60%|██████    | 1804/2984 [5:10:49<3:19:10, 10.13s/it]
 60%|██████    | 1805/2984 [5:10:59<3:18:53, 10.12s/it]
 61%|██████    | 1806/2984 [5:11:09<3:18:46, 10.12s/it]
 61%|██████    | 1807/2984 [5:11:19<3:18:29, 10.12s/it]
 61%|██████    | 1808/2984 [5:11:29<3:18:26, 10.12s/it]
 61%|██████    | 1809/2984 [5:11:39<3:18:15, 10.12s/it]
 61%|███�
0: {'loss': 0.5309, 'grad_norm': 1.0120464494739698, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.36}
0: {'loss': 0.5124, 'grad_norm': 1.2198659724624672, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.37}
0: ��██    | 1810/2984 [5:11:49<3:18:17, 10.13s/it]
                                                       

 61%|██████    | 1810/2984 [5:11:49<3:18:17, 10.13s/it]
 61%|██████    | 1811/2984 [5:12:00<3:18:35, 10.16s/it]
 61%|██████    | 1812/2984 [5:12:10<3:18:31, 10.16s/it]
 61%|██████    | 1813/2984 [5:12:20<3:18:41, 10.18s/it]
 61%|██████    | 1814/2984 [5:12:30<3:18:41, 10.19s/it]
 61%|██████    | 1815/2984 [5:12:40<3:18:33, 10.19s/it]
 61%|██████    | 1816/2984 [5:12:51<3:19:49, 10.26s/it]
 61%|██████    | 1817/2984 [5:13:01<3:20:18, 10.30s/it]
 61%|██████    | 1818/2984 [5:13:11<3:19:36, 10.27s/it]
 61%|██████    | 1819/2984 [5:13:22<3:19:11, 10.26s/it]
 61%|██████    | 1820/2984 [5:13:32<3:19:06, 10.26s/it]
                                                       

 61%|██████    | 1820/2984 [5:13:32<3:19:06, 10.26s/it]
 61%|██████    | 1821/2984 [5
0: {'loss': 0.5118, 'grad_norm': 1.069871372454056, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.37}
0: :13:42<3:18:54, 10.26s/it]
 61%|██████    | 1822/2984 [5:13:52<3:18:24, 10.24s/it]
 61%|██████    | 1823/2984 [5:14:03<3:18:07, 10.24s/it]
 61%|██████    | 1824/2984 [5:14:13<3:17:39, 10.22s/it]
 61%|██████    | 1825/2984 [5:14:23<3:17:18, 10.21s/it]
 61%|██████    | 1826/2984 [5:14:33<3:16:35, 10.19s/it]
 61%|██████    | 1827/2984 [5:14:43<3:16:20, 10.18s/it]
 61%|██████▏   | 1828/2984 [5:14:54<3:16:23, 10.19s/it]
 61%|██████▏   | 1829/2984 [5:15:04<3:16:12, 10.19s/it]
 61%|██████▏   | 1830/2984 [5:15:14<3:16:09, 10.20s/it]
                                                       

 61%|██████▏   | 1830/2984 [5:15:14<3:16:09, 10.20s/it]
 61%|██████▏   | 1831/2984 [5:15:24<3:16:03, 10.20s/it]
 61%|██████▏   | 1832/2984 [5:15:34<3:15:56, 10.21s/it]
 61%|██████▏   | 1833/2984 [5:15:45<3:15:46, 10.21s/it]
 61%|██████▏   | 1834/2984 [
0: {'loss': 0.5308, 'grad_norm': 0.9165999242803844, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.37}
0: 5:15:55<3:15:08, 10.18s/it]
 61%|██████▏   | 1835/2984 [5:16:05<3:15:16, 10.20s/it]
 62%|██████▏   | 1836/2984 [5:16:15<3:15:04, 10.20s/it]
 62%|██████▏   | 1837/2984 [5:16:25<3:14:36, 10.18s/it]
 62%|██████▏   | 1838/2984 [5:16:36<3:14:33, 10.19s/it]
 62%|██████▏   | 1839/2984 [5:16:46<3:14:55, 10.21s/it]
 62%|██████▏   | 1840/2984 [5:16:56<3:15:28, 10.25s/it]
                                                       

 62%|██████▏   | 1840/2984 [5:16:56<3:15:28, 10.25s/it]
 62%|██████▏   | 1841/2984 [5:17:06<3:15:53, 10.28s/it]
 62%|██████▏   | 1842/2984 [5:17:17<3:16:16, 10.31s/it]
 62%|██████▏   | 1843/2984 [5:17:27<3:16:16, 10.32s/it]
 62%|██████▏   | 1844/2984 [5:17:38<3:16:28, 10.34s/it]
 62%|██████▏   | 1845/2984 [5:17:48<3:16:30, 10.35s/it]
 62%|██████▏   | 1846/2984 [5:17:58<3:16:27, 10.36s/it]
 62%|██████▏   
0: {'loss': 0.5141, 'grad_norm': 1.0250354197220282, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.37}
0: | 1847/2984 [5:18:09<3:17:00, 10.40s/it]
 62%|██████▏   | 1848/2984 [5:18:19<3:15:49, 10.34s/it]
 62%|██████▏   | 1849/2984 [5:18:29<3:14:41, 10.29s/it]
 62%|██████▏   | 1850/2984 [5:18:39<3:13:50, 10.26s/it]
                                                       

 62%|██████▏   | 1850/2984 [5:18:39<3:13:50, 10.26s/it]
 62%|██████▏   | 1851/2984 [5:18:49<3:12:45, 10.21s/it]
 62%|██████▏   | 1852/2984 [5:19:00<3:11:54, 10.17s/it]
 62%|██████▏   | 1853/2984 [5:19:10<3:11:44, 10.17s/it]
 62%|██████▏   | 1854/2984 [5:19:20<3:11:31, 10.17s/it]
 62%|██████▏   | 1855/2984 [5:19:30<3:10:55, 10.15s/it]
 62%|██████▏   | 1856/2984 [5:19:40<3:10:50, 10.15s/it]
 62%|██████▏   | 1857/2984 [5:19:50<3:10:40, 10.15s/it]
 62%|██████▏   | 1858/2984 [5:20:00<3:10:17, 10.14s/it]
 62%|██████▏   | 1859/2984 [5:20:11<3:10:22, 10.15s/it]
 62%|███�
0: {'loss': 0.5072, 'grad_norm': 0.8767713425905129, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.37}
0: {'loss': 0.5068, 'grad_norm': 0.9416141039295927, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.38}
0: �██▏   | 1860/2984 [5:20:21<3:10:39, 10.18s/it]
                                                       

 62%|██████▏   | 1860/2984 [5:20:21<3:10:39, 10.18s/it]
 62%|██████▏   | 1861/2984 [5:20:31<3:11:28, 10.23s/it]
 62%|██████▏   | 1862/2984 [5:20:42<3:12:17, 10.28s/it]
 62%|██████▏   | 1863/2984 [5:20:52<3:12:14, 10.29s/it]
 62%|██████▏   | 1864/2984 [5:21:02<3:12:11, 10.30s/it]
 62%|██████▎   | 1865/2984 [5:21:13<3:12:16, 10.31s/it]
 63%|██████▎   | 1866/2984 [5:21:23<3:12:17, 10.32s/it]
 63%|██████▎   | 1867/2984 [5:21:33<3:11:50, 10.30s/it]
 63%|██████▎   | 1868/2984 [5:21:44<3:12:05, 10.33s/it]
 63%|██████▎   | 1869/2984 [5:21:54<3:12:17, 10.35s/it]
 63%|██████▎   | 1870/2984 [5:22:04<3:12:35, 10.37s/it]
                                                       

 63%|██████▎   | 1870/2984 [5:22:04<3:12:35, 10.37s/it]
 63%|███�
0: {'loss': 0.5014, 'grad_norm': 1.0382102867575973, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.38}
0: �██▎   | 1871/2984 [5:22:15<3:12:06, 10.36s/it]
 63%|██████▎   | 1872/2984 [5:22:25<3:12:02, 10.36s/it]
 63%|██████▎   | 1873/2984 [5:22:35<3:11:51, 10.36s/it]
 63%|██████▎   | 1874/2984 [5:22:46<3:11:03, 10.33s/it]
 63%|██████▎   | 1875/2984 [5:22:56<3:10:30, 10.31s/it]
 63%|██████▎   | 1876/2984 [5:23:06<3:10:12, 10.30s/it]
 63%|██████▎   | 1877/2984 [5:23:17<3:10:24, 10.32s/it]
 63%|██████▎   | 1878/2984 [5:23:27<3:10:29, 10.33s/it]
 63%|██████▎   | 1879/2984 [5:23:37<3:10:29, 10.34s/it]
 63%|██████▎   | 1880/2984 [5:23:48<3:09:26, 10.30s/it]
                                                       

 63%|██████▎   | 1880/2984 [5:23:48<3:09:26, 10.30s/it]
 63%|██████▎   | 1881/2984 [5:23:58<3:08:40, 10.26s/it]
 63%|██████▎   | 1882/2984 [5:24:08<3:07:41, 10.22s/it]
 63%|██████▎   | 1883/2984 [5:24:18<3:07:26, 10.21s/it]
 63
0: {'loss': 0.4986, 'grad_norm': 0.9424455449506925, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.38}
0: %|██████▎   | 1884/2984 [5:24:28<3:07:02, 10.20s/it]
 63%|██████▎   | 1885/2984 [5:24:38<3:07:23, 10.23s/it]
 63%|██████▎   | 1886/2984 [5:24:49<3:06:38, 10.20s/it]
 63%|██████▎   | 1887/2984 [5:24:59<3:05:45, 10.16s/it]
 63%|██████▎   | 1888/2984 [5:25:09<3:05:07, 10.13s/it]
 63%|██████▎   | 1889/2984 [5:25:19<3:04:47, 10.13s/it]
 63%|██████▎   | 1890/2984 [5:25:29<3:04:46, 10.13s/it]
                                                       

 63%|██████▎   | 1890/2984 [5:25:29<3:04:46, 10.13s/it]
 63%|██████▎   | 1891/2984 [5:25:39<3:05:07, 10.16s/it]
 63%|██████▎   | 1892/2984 [5:25:49<3:05:15, 10.18s/it]
 63%|██████▎   | 1893/2984 [5:26:00<3:05:12, 10.19s/it]
 63%|██████▎   | 1894/2984 [5:26:10<3:04:55, 10.18s/it]
 64%|██████▎   | 1895/2984 [5:26:20<3:04:45, 10.18s/it]
 64%|██████▎   | 1896/2984 [5:26:30<3:04:35, 1
0: {'loss': 0.5056, 'grad_norm': 0.9326290657730396, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.38}
0: 0.18s/it]
 64%|██████▎   | 1897/2984 [5:26:40<3:04:09, 10.17s/it]
 64%|██████▎   | 1898/2984 [5:26:50<3:03:50, 10.16s/it]
 64%|██████▎   | 1899/2984 [5:27:01<3:04:54, 10.23s/it]
 64%|██████▎   | 1900/2984 [5:27:11<3:06:19, 10.31s/it]
                                                       

 64%|██████▎   | 1900/2984 [5:27:11<3:06:19, 10.31s/it]
 64%|██████▎   | 1901/2984 [5:27:22<3:07:32, 10.39s/it]
 64%|██████▎   | 1902/2984 [5:27:32<3:06:43, 10.35s/it]
 64%|██████▍   | 1903/2984 [5:27:43<3:06:36, 10.36s/it]
 64%|██████▍   | 1904/2984 [5:27:53<3:06:40, 10.37s/it]
 64%|██████▍   | 1905/2984 [5:28:03<3:06:41, 10.38s/it]
 64%|██████▍   | 1906/2984 [5:28:14<3:06:05, 10.36s/it]
 64%|██████▍   | 1907/2984 [5:28:24<3:05:27, 10.33s/it]
 64%|██████▍   | 1908/2984 [5:28:34<3:05:26, 10.34s/it]
 64%|██████▍   | 1909/2984 [5:28:
0: {'loss': 0.505, 'grad_norm': 0.8504492047752286, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.38}
0: {'loss': 0.5003, 'grad_norm': 0.960998472972023, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.39}
0: 45<3:05:31, 10.35s/it]
 64%|██████▍   | 1910/2984 [5:28:55<3:05:40, 10.37s/it]
                                                       

 64%|██████▍   | 1910/2984 [5:28:55<3:05:40, 10.37s/it]
 64%|██████▍   | 1911/2984 [5:29:05<3:05:28, 10.37s/it]
 64%|██████▍   | 1912/2984 [5:29:16<3:04:47, 10.34s/it]
 64%|██████▍   | 1913/2984 [5:29:26<3:04:52, 10.36s/it]
 64%|██████▍   | 1914/2984 [5:29:37<3:05:07, 10.38s/it]
 64%|██████▍   | 1915/2984 [5:29:47<3:05:14, 10.40s/it]
 64%|██████▍   | 1916/2984 [5:29:58<3:05:41, 10.43s/it]
 64%|██████▍   | 1917/2984 [5:30:08<3:04:54, 10.40s/it]
 64%|██████▍   | 1918/2984 [5:30:18<3:04:07, 10.36s/it]
 64%|██████▍   | 1919/2984 [5:30:28<3:03:06, 10.32s/it]
 64%|██████▍   | 1920/2984 [5:30:39<3:02:10, 10.27s/it]
                                                       

 64%|██████▍   | 1920/2984 [5:30:
0: {'loss': 0.4984, 'grad_norm': 0.8787774240040781, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.39}
0: 39<3:02:10, 10.27s/it]
 64%|██████▍   | 1921/2984 [5:30:49<3:01:49, 10.26s/it]
 64%|██████▍   | 1922/2984 [5:30:59<3:00:50, 10.22s/it]
 64%|██████▍   | 1923/2984 [5:31:09<3:00:30, 10.21s/it]
 64%|██████▍   | 1924/2984 [5:31:19<3:01:22, 10.27s/it]
 65%|██████▍   | 1925/2984 [5:31:30<3:02:24, 10.34s/it]
 65%|██████▍   | 1926/2984 [5:31:40<3:01:53, 10.32s/it]
 65%|██████▍   | 1927/2984 [5:31:50<3:00:56, 10.27s/it]
 65%|██████▍   | 1928/2984 [5:32:01<3:00:09, 10.24s/it]
 65%|██████▍   | 1929/2984 [5:32:11<2:59:16, 10.20s/it]
 65%|██████▍   | 1930/2984 [5:32:21<2:59:11, 10.20s/it]
                                                       

 65%|██████▍   | 1930/2984 [5:32:21<2:59:11, 10.20s/it]
 65%|██████▍   | 1931/2984 [5:32:31<2:58:59, 10.20s/it]
 65%|██████▍   | 1932/2984 [5:32:41<2:58:46, 10.20s/it]
 65%|██████▍   | 193
0: {'loss': 0.4988, 'grad_norm': 0.9556225070689345, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.39}
0: 3/2984 [5:32:51<2:57:54, 10.16s/it]
 65%|██████▍   | 1934/2984 [5:33:01<2:57:19, 10.13s/it]
 65%|██████▍   | 1935/2984 [5:33:12<2:57:24, 10.15s/it]
 65%|██████▍   | 1936/2984 [5:33:22<2:57:11, 10.14s/it]
 65%|██████▍   | 1937/2984 [5:33:32<2:57:14, 10.16s/it]
 65%|██████▍   | 1938/2984 [5:33:42<2:57:24, 10.18s/it]
 65%|██████▍   | 1939/2984 [5:33:52<2:57:30, 10.19s/it]
 65%|██████▌   | 1940/2984 [5:34:03<2:57:26, 10.20s/it]
                                                       

 65%|██████▌   | 1940/2984 [5:34:03<2:57:26, 10.20s/it]
 65%|██████▌   | 1941/2984 [5:34:13<2:57:26, 10.21s/it]
 65%|██████▌   | 1942/2984 [5:34:23<2:56:41, 10.17s/it]
 65%|██████▌   | 1943/2984 [5:34:33<2:56:31, 10.17s/it]
 65%|██████▌   | 1944/2984 [5:34:43<2:56:06, 10.16s/it]
 65%|██████▌   | 1945/2984 [5:34:53<2:55:46, 10.15s/it]
 65%|█████�
0: {'loss': 0.4996, 'grad_norm': 0.962384477469165, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.39}
0: ��▌   | 1946/2984 [5:35:03<2:55:27, 10.14s/it]
 65%|██████▌   | 1947/2984 [5:35:14<2:55:20, 10.15s/it]
 65%|██████▌   | 1948/2984 [5:35:24<2:55:01, 10.14s/it]
 65%|██████▌   | 1949/2984 [5:35:34<2:55:20, 10.16s/it]
 65%|██████▌   | 1950/2984 [5:35:44<2:55:47, 10.20s/it]
                                                       

 65%|██████▌   | 1950/2984 [5:35:44<2:55:47, 10.20s/it]
 65%|██████▌   | 1951/2984 [5:35:54<2:55:37, 10.20s/it]
 65%|██████▌   | 1952/2984 [5:36:05<2:55:30, 10.20s/it]
 65%|██████▌   | 1953/2984 [5:36:15<2:55:16, 10.20s/it]
 65%|██████▌   | 1954/2984 [5:36:25<2:55:03, 10.20s/it]
 66%|██████▌   | 1955/2984 [5:36:35<2:55:01, 10.21s/it]
 66%|██████▌   | 1956/2984 [5:36:45<2:54:51, 10.21s/it]
 66%|██████▌   | 1957/2984 [5:36:56<2:54:50, 10.21s/it]
 66%|██████▌   | 1958/2984 [5:37:06<2:54:50, 10.23s/it]
 66%|█
0: {'loss': 0.5129, 'grad_norm': 1.2225026981603808, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.39}
0: {'loss': 0.5254, 'grad_norm': 0.9135982574191678, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.4}
0: █████▌   | 1959/2984 [5:37:16<2:54:47, 10.23s/it]
 66%|██████▌   | 1960/2984 [5:37:27<2:55:39, 10.29s/it]
                                                       

 66%|██████▌   | 1960/2984 [5:37:27<2:55:39, 10.29s/it]
 66%|██████▌   | 1961/2984 [5:37:37<2:55:37, 10.30s/it]
 66%|██████▌   | 1962/2984 [5:37:47<2:55:16, 10.29s/it]
 66%|██████▌   | 1963/2984 [5:37:58<2:55:17, 10.30s/it]
 66%|██████▌   | 1964/2984 [5:38:08<2:54:59, 10.29s/it]
 66%|██████▌   | 1965/2984 [5:38:18<2:55:01, 10.31s/it]
 66%|██████▌   | 1966/2984 [5:38:29<2:55:23, 10.34s/it]
 66%|██████▌   | 1967/2984 [5:38:39<2:55:58, 10.38s/it]
 66%|██████▌   | 1968/2984 [5:38:49<2:55:46, 10.38s/it]
 66%|██████▌   | 1969/2984 [5:39:00<2:56:06, 10.41s/it]
 66%|██████▌   | 1970/2984 [5:39:10<2:55:54, 10.41s/it]
                                                       

 66%|█
0: {'loss': 0.5139, 'grad_norm': 0.93147576017722, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.4}
0: █████▌   | 1970/2984 [5:39:10<2:55:54, 10.41s/it]
 66%|██████▌   | 1971/2984 [5:39:21<2:56:03, 10.43s/it]
 66%|██████▌   | 1972/2984 [5:39:31<2:56:04, 10.44s/it]
 66%|██████▌   | 1973/2984 [5:39:42<2:56:09, 10.45s/it]
 66%|██████▌   | 1974/2984 [5:39:52<2:55:54, 10.45s/it]
 66%|██████▌   | 1975/2984 [5:40:02<2:54:51, 10.40s/it]
 66%|██████▌   | 1976/2984 [5:40:13<2:54:06, 10.36s/it]
 66%|██████▋   | 1977/2984 [5:40:23<2:53:33, 10.34s/it]
 66%|██████▋   | 1978/2984 [5:40:33<2:52:37, 10.30s/it]
 66%|██████▋   | 1979/2984 [5:40:43<2:51:51, 10.26s/it]
 66%|██████▋   | 1980/2984 [5:40:54<2:51:46, 10.27s/it]
                                                       

 66%|██████▋   | 1980/2984 [5:40:54<2:51:46, 10.27s/it]
 66%|██████▋   | 1981/2984 [5:41:04<2:51:42, 10.27s/it]
 66%|██████▋   | 1982/2984 [5:41:14<2:51:39, 10.28s
0: {'loss': 0.5158, 'grad_norm': 0.8579680207861539, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.4}
0: /it]
 66%|██████▋   | 1983/2984 [5:41:25<2:52:00, 10.31s/it]
 66%|██████▋   | 1984/2984 [5:41:35<2:52:15, 10.34s/it]
 67%|██████▋   | 1985/2984 [5:41:46<2:53:31, 10.42s/it]
 67%|██████▋   | 1986/2984 [5:41:56<2:53:00, 10.40s/it]
 67%|██████▋   | 1987/2984 [5:42:07<2:53:44, 10.46s/it]
 67%|██████▋   | 1988/2984 [5:42:17<2:52:20, 10.38s/it]
 67%|██████▋   | 1989/2984 [5:42:27<2:50:29, 10.28s/it]
 67%|██████▋   | 1990/2984 [5:42:37<2:49:31, 10.23s/it]
                                                       

 67%|██████▋   | 1990/2984 [5:42:37<2:49:31, 10.23s/it]
 67%|██████▋   | 1991/2984 [5:42:47<2:49:11, 10.22s/it]
 67%|██████▋   | 1992/2984 [5:42:57<2:48:49, 10.21s/it]
 67%|██████▋   | 1993/2984 [5:43:08<2:49:22, 10.25s/it]
 67%|██████▋   | 1994/2984 [5:43:18<2:48:38, 10.22s/it]
 67%|██████▋   | 1995/2984 [5:43:28<2:
0: {'loss': 0.5041, 'grad_norm': 0.9560698047937931, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.4}
0: 48:13, 10.21s/it]
 67%|██████▋   | 1996/2984 [5:43:38<2:47:55, 10.20s/it]
 67%|██████▋   | 1997/2984 [5:43:48<2:47:24, 10.18s/it]
 67%|██████▋   | 1998/2984 [5:43:58<2:46:57, 10.16s/it]
 67%|██████▋   | 1999/2984 [5:44:08<2:46:21, 10.13s/it]
 67%|██████▋   | 2000/2984 [5:44:19<2:46:41, 10.16s/it]
                                                       

 67%|██████▋   | 2000/2984 [5:44:19<2:46:41, 10.16s/it]
 67%|██████▋   | 2001/2984 [5:44:29<2:47:56, 10.25s/it]
 67%|██████▋   | 2002/2984 [5:44:40<2:49:21, 10.35s/it]
 67%|██████▋   | 2003/2984 [5:44:50<2:50:07, 10.40s/it]
 67%|██████▋   | 2004/2984 [5:45:01<2:50:10, 10.42s/it]
 67%|██████▋   | 2005/2984 [5:45:11<2:49:28, 10.39s/it]
 67%|██████▋   | 2006/2984 [5:45:21<2:49:35, 10.40s/it]
 67%|██████▋   | 2007/2984 [5:45:32<2:49:36, 10.42s/it]
 67%|██████▋   | 2008/298
0: {'loss': 0.4932, 'grad_norm': 0.9387657675261316, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.4}
0: 4 [5:45:42<2:50:04, 10.46s/it]
 67%|██████▋   | 2009/2984 [5:45:53<2:49:36, 10.44s/it]
 67%|██████▋   | 2010/2984 [5:46:03<2:48:57, 10.41s/it]
                                                       

 67%|██████▋   | 2010/2984 [5:46:03<2:48:57, 10.41s/it]
 67%|██████▋   | 2011/2984 [5:46:13<2:48:01, 10.36s/it]
 67%|██████▋   | 2012/2984 [5:46:24<2:47:50, 10.36s/it]
 67%|██████▋   | 2013/2984 [5:46:34<2:47:32, 10.35s/it]
 67%|██████▋   | 2014/2984 [5:46:44<2:47:07, 10.34s/it]
 68%|██████▊   | 2015/2984 [5:46:55<2:46:56, 10.34s/it]
 68%|██████▊   | 2016/2984 [5:47:05<2:47:03, 10.35s/it]
 68%|██████▊   | 2017/2984 [5:47:16<2:46:59, 10.36s/it]
 68%|██████▊   | 2018/2984 [5:47:26<2:47:10, 10.38s/it]
 68%|██████▊   | 2019/2984 [5:47:36<2:47:16, 10.40s/it]
 68%|██████▊   | 2020/2984 [5:47:47<2:47:30, 10.43s/it]
                          
0: {'loss': 0.5082, 'grad_norm': 0.9265447872946807, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.41}
0: {'loss': 0.5089, 'grad_norm': 0.9782595781086991, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.41}
0:                              

 68%|██████▊   | 2020/2984 [5:47:47<2:47:30, 10.43s/it]
 68%|██████▊   | 2021/2984 [5:47:57<2:47:06, 10.41s/it]
 68%|██████▊   | 2022/2984 [5:48:07<2:45:47, 10.34s/it]
 68%|██████▊   | 2023/2984 [5:48:18<2:44:33, 10.27s/it]
 68%|██████▊   | 2024/2984 [5:48:28<2:44:00, 10.25s/it]
 68%|██████▊   | 2025/2984 [5:48:38<2:43:51, 10.25s/it]
 68%|██████▊   | 2026/2984 [5:48:48<2:43:24, 10.23s/it]
 68%|██████▊   | 2027/2984 [5:48:58<2:43:04, 10.22s/it]
 68%|██████▊   | 2028/2984 [5:49:09<2:42:42, 10.21s/it]
 68%|██████▊   | 2029/2984 [5:49:19<2:42:24, 10.20s/it]
 68%|██████▊   | 2030/2984 [5:49:29<2:42:32, 10.22s/it]
                                                       

 68%|██████▊   | 2030/2984 [5:49:29<2:42:32, 10.22s/it]
 68%|██████▊   | 2031/2984 [5:49:39<2:42:41, 10.24s/it]
 68%|██████▊
0: {'loss': 0.5047, 'grad_norm': 0.9094120453348998, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.41}
0:    | 2032/2984 [5:49:50<2:43:00, 10.27s/it]
 68%|██████▊   | 2033/2984 [5:50:00<2:43:00, 10.28s/it]
 68%|██████▊   | 2034/2984 [5:50:10<2:43:00, 10.30s/it]
 68%|██████▊   | 2035/2984 [5:50:21<2:43:46, 10.36s/it]
 68%|██████▊   | 2036/2984 [5:50:31<2:43:08, 10.33s/it]
 68%|██████▊   | 2037/2984 [5:50:41<2:43:01, 10.33s/it]
 68%|██████▊   | 2038/2984 [5:50:52<2:43:20, 10.36s/it]
 68%|██████▊   | 2039/2984 [5:51:02<2:42:45, 10.33s/it]
 68%|██████▊   | 2040/2984 [5:51:12<2:42:29, 10.33s/it]
                                                       

 68%|██████▊   | 2040/2984 [5:51:12<2:42:29, 10.33s/it]
 68%|██████▊   | 2041/2984 [5:51:23<2:42:11, 10.32s/it]
 68%|██████▊   | 2042/2984 [5:51:33<2:41:59, 10.32s/it]
 68%|██████▊   | 2043/2984 [5:51:43<2:41:34, 10.30s/it]
 68%|██████▊   | 2044/2984 [5:51:54<2:41:20, 10.30s/it]
 69%|██�
0: {'loss': 0.516, 'grad_norm': 0.8928339699557475, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.41}
0: �███▊   | 2045/2984 [5:52:04<2:40:51, 10.28s/it]
 69%|██████▊   | 2046/2984 [5:52:14<2:40:19, 10.26s/it]
 69%|██████▊   | 2047/2984 [5:52:24<2:39:47, 10.23s/it]
 69%|██████▊   | 2048/2984 [5:52:34<2:38:42, 10.17s/it]
 69%|██████▊   | 2049/2984 [5:52:44<2:38:09, 10.15s/it]
 69%|██████▊   | 2050/2984 [5:52:55<2:38:42, 10.19s/it]
                                                       

 69%|██████▊   | 2050/2984 [5:52:55<2:38:42, 10.19s/it]
 69%|██████▊   | 2051/2984 [5:53:05<2:41:15, 10.37s/it]
 69%|██████▉   | 2052/2984 [5:53:15<2:39:33, 10.27s/it]
 69%|██████▉   | 2053/2984 [5:53:26<2:38:41, 10.23s/it]
 69%|██████▉   | 2054/2984 [5:53:36<2:38:19, 10.21s/it]
 69%|██████▉   | 2055/2984 [5:53:46<2:38:17, 10.22s/it]
 69%|██████▉   | 2056/2984 [5:53:56<2:37:40, 10.19s/it]
 69%|██████▉   | 2057/2984 [5:54:06<2:37:18, 10.18s/it]
0: {'loss': 0.5116, 'grad_norm': 0.8946434932611803, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.41}
0:  69%|██████▉   | 2058/2984 [5:54:16<2:36:51, 10.16s/it]
 69%|██████▉   | 2059/2984 [5:54:27<2:36:14, 10.13s/it]
 69%|██████▉   | 2060/2984 [5:54:37<2:35:57, 10.13s/it]
                                                       

 69%|██████▉   | 2060/2984 [5:54:37<2:35:57, 10.13s/it]
 69%|██████▉   | 2061/2984 [5:54:47<2:35:55, 10.14s/it]
 69%|██████▉   | 2062/2984 [5:54:57<2:36:46, 10.20s/it]
 69%|██████▉   | 2063/2984 [5:55:07<2:36:08, 10.17s/it]
 69%|██████▉   | 2064/2984 [5:55:17<2:35:37, 10.15s/it]
 69%|██████▉   | 2065/2984 [5:55:27<2:35:24, 10.15s/it]
 69%|██████▉   | 2066/2984 [5:55:38<2:35:18, 10.15s/it]
 69%|██████▉   | 2067/2984 [5:55:48<2:35:19, 10.16s/it]
 69%|██████▉   | 2068/2984 [5:55:58<2:35:04, 10.16s/it]
 69%|██████▉   | 2069/2984 [5:56:08<2:34:54, 10.16s/it]
 69%|██████▉   | 2070/2984 [5:56:18<2:35:12
0: {'loss': 0.5207, 'grad_norm': 0.8954473557055929, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.42}
0: {'loss': 0.513, 'grad_norm': 0.8739812189154775, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.42}
0: , 10.19s/it]
                                                       

 69%|██████▉   | 2070/2984 [5:56:18<2:35:12, 10.19s/it]
 69%|██████▉   | 2071/2984 [5:56:29<2:35:05, 10.19s/it]
 69%|██████▉   | 2072/2984 [5:56:39<2:34:24, 10.16s/it]
 69%|██████▉   | 2073/2984 [5:56:49<2:34:19, 10.16s/it]
 70%|██████▉   | 2074/2984 [5:56:59<2:33:41, 10.13s/it]
 70%|██████▉   | 2075/2984 [5:57:09<2:33:20, 10.12s/it]
 70%|██████▉   | 2076/2984 [5:57:19<2:32:20, 10.07s/it]
 70%|██████▉   | 2077/2984 [5:57:29<2:31:47, 10.04s/it]
 70%|██████▉   | 2078/2984 [5:57:39<2:31:51, 10.06s/it]
 70%|██████▉   | 2079/2984 [5:57:49<2:30:56, 10.01s/it]
 70%|██████▉   | 2080/2984 [5:57:59<2:29:57,  9.95s/it]
                                                       

 70%|██████▉   | 2080/2984 [5:57:59<2:29:57,  9.95s/it]
 70%|██████▉   | 2081/2984 [5:58:09<2:29:33
0: {'loss': 0.5072, 'grad_norm': 1.2220841533679818, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.42}
0: ,  9.94s/it]
 70%|██████▉   | 2082/2984 [5:58:19<2:29:55,  9.97s/it]
 70%|██████▉   | 2083/2984 [5:58:29<2:30:06, 10.00s/it]
 70%|██████▉   | 2084/2984 [5:58:39<2:30:20, 10.02s/it]
 70%|██████▉   | 2085/2984 [5:58:49<2:30:31, 10.05s/it]
 70%|██████▉   | 2086/2984 [5:58:59<2:30:36, 10.06s/it]
 70%|██████▉   | 2087/2984 [5:59:09<2:30:47, 10.09s/it]
 70%|██████▉   | 2088/2984 [5:59:19<2:31:06, 10.12s/it]
 70%|███████   | 2089/2984 [5:59:29<2:30:57, 10.12s/it]
 70%|███████   | 2090/2984 [5:59:39<2:30:08, 10.08s/it]
                                                       

 70%|███████   | 2090/2984 [5:59:39<2:30:08, 10.08s/it]
 70%|███████   | 2091/2984 [5:59:50<2:30:24, 10.11s/it]
 70%|███████   | 2092/2984 [6:00:00<2:30:55, 10.15s/it]
 70%|███████   | 2093/2984 [6:00:10<2:31:09, 10.18s/it]
 70%|███████   | 2094/2984 [6:
0: {'loss': 0.5201, 'grad_norm': 0.9102800000916964, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.42}
0: 00:20<2:30:53, 10.17s/it]
 70%|███████   | 2095/2984 [6:00:30<2:30:25, 10.15s/it]
 70%|███████   | 2096/2984 [6:00:40<2:29:50, 10.12s/it]
 70%|███████   | 2097/2984 [6:00:51<2:29:22, 10.10s/it]
 70%|███████   | 2098/2984 [6:01:01<2:29:05, 10.10s/it]
 70%|███████   | 2099/2984 [6:01:11<2:28:44, 10.08s/it]
 70%|███████   | 2100/2984 [6:01:21<2:28:41, 10.09s/it]
                                                       

 70%|███████   | 2100/2984 [6:01:21<2:28:41, 10.09s/it]
 70%|███████   | 2101/2984 [6:01:31<2:28:35, 10.10s/it]
 70%|███████   | 2102/2984 [6:01:41<2:28:31, 10.10s/it]
 70%|███████   | 2103/2984 [6:01:51<2:28:11, 10.09s/it]
 71%|███████   | 2104/2984 [6:02:01<2:28:02, 10.09s/it]
 71%|███████   | 2105/2984 [6:02:11<2:27:54, 10.10s/it]
 71%|███████   | 2106/2984 [6:02:21<2:27:37, 10.09s/it]
 71%|███████   | 
0: {'loss': 0.5125, 'grad_norm': 0.8509085087101516, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.42}
0: 2107/2984 [6:02:31<2:27:26, 10.09s/it]
 71%|███████   | 2108/2984 [6:02:41<2:27:01, 10.07s/it]
 71%|███████   | 2109/2984 [6:02:52<2:26:51, 10.07s/it]
 71%|███████   | 2110/2984 [6:03:02<2:26:29, 10.06s/it]
                                                       

 71%|███████   | 2110/2984 [6:03:02<2:26:29, 10.06s/it]
 71%|███████   | 2111/2984 [6:03:12<2:25:54, 10.03s/it]
 71%|███████   | 2112/2984 [6:03:22<2:25:40, 10.02s/it]
 71%|███████   | 2113/2984 [6:03:32<2:25:51, 10.05s/it]
 71%|███████   | 2114/2984 [6:03:42<2:25:20, 10.02s/it]
 71%|███████   | 2115/2984 [6:03:52<2:25:10, 10.02s/it]
 71%|███████   | 2116/2984 [6:04:02<2:25:04, 10.03s/it]
 71%|███████   | 2117/2984 [6:04:12<2:24:49, 10.02s/it]
 71%|███████   | 2118/2984 [6:04:22<2:24:34, 10.02s/it]
 71%|███████   | 2119/2984 [6:04:32<2:24:15, 10.01s/it]
 71%|████�
0: {'loss': 0.5036, 'grad_norm': 1.2359610504014933, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.43}
0: {'loss': 0.4922, 'grad_norm': 0.9037564144017687, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.43}
0: ��██   | 2120/2984 [6:04:42<2:24:13, 10.02s/it]
                                                       

 71%|███████   | 2120/2984 [6:04:42<2:24:13, 10.02s/it]
 71%|███████   | 2121/2984 [6:04:52<2:24:06, 10.02s/it]
 71%|███████   | 2122/2984 [6:05:02<2:23:57, 10.02s/it]
 71%|███████   | 2123/2984 [6:05:12<2:23:37, 10.01s/it]
 71%|███████   | 2124/2984 [6:05:22<2:23:29, 10.01s/it]
 71%|███████   | 2125/2984 [6:05:32<2:23:12, 10.00s/it]
 71%|███████   | 2126/2984 [6:05:42<2:22:46,  9.98s/it]
 71%|███████▏  | 2127/2984 [6:05:52<2:22:49, 10.00s/it]
 71%|███████▏  | 2128/2984 [6:06:02<2:22:54, 10.02s/it]
 71%|███████▏  | 2129/2984 [6:06:12<2:22:34, 10.01s/it]
 71%|███████▏  | 2130/2984 [6:06:22<2:22:21, 10.00s/it]
                                                       

 71%|███████▏  | 2130/2984 [6:06:22<2:22:21, 10.00s/it]
 71%|█
0: {'loss': 0.5089, 'grad_norm': 0.9203400260646102, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.43}
0: ██████▏  | 2131/2984 [6:06:32<2:22:02,  9.99s/it]
 71%|███████▏  | 2132/2984 [6:06:42<2:22:02, 10.00s/it]
 71%|███████▏  | 2133/2984 [6:06:52<2:21:44,  9.99s/it]
 72%|███████▏  | 2134/2984 [6:07:02<2:21:46, 10.01s/it]
 72%|███████▏  | 2135/2984 [6:07:12<2:21:44, 10.02s/it]
 72%|███████▏  | 2136/2984 [6:07:22<2:21:29, 10.01s/it]
 72%|███████▏  | 2137/2984 [6:07:32<2:21:30, 10.02s/it]
 72%|███████▏  | 2138/2984 [6:07:42<2:21:11, 10.01s/it]
 72%|███████▏  | 2139/2984 [6:07:52<2:21:01, 10.01s/it]
 72%|███████▏  | 2140/2984 [6:08:02<2:20:56, 10.02s/it]
                                                       

 72%|███████▏  | 2140/2984 [6:08:02<2:20:56, 10.02s/it]
 72%|███████▏  | 2141/2984 [6:08:12<2:21:00, 10.04s/it]
 72%|███████▏  | 2142/2984 [6:08:22<2:20:57, 10.04s/it]
 72%|███████▏  | 2143/2
0: {'loss': 0.5029, 'grad_norm': 0.8062518039551, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.43}
0: 984 [6:08:32<2:20:44, 10.04s/it]
 72%|███████▏  | 2144/2984 [6:08:42<2:20:42, 10.05s/it]
 72%|███████▏  | 2145/2984 [6:08:52<2:20:46, 10.07s/it]
 72%|███████▏  | 2146/2984 [6:09:02<2:20:37, 10.07s/it]
 72%|███████▏  | 2147/2984 [6:09:12<2:20:36, 10.08s/it]
 72%|███████▏  | 2148/2984 [6:09:22<2:20:22, 10.07s/it]
 72%|███████▏  | 2149/2984 [6:09:33<2:21:12, 10.15s/it]
 72%|███████▏  | 2150/2984 [6:09:43<2:22:09, 10.23s/it]
                                                       

 72%|███████▏  | 2150/2984 [6:09:43<2:22:09, 10.23s/it]
 72%|███████▏  | 2151/2984 [6:09:53<2:21:42, 10.21s/it]
 72%|███████▏  | 2152/2984 [6:10:03<2:20:56, 10.16s/it]
 72%|███████▏  | 2153/2984 [6:10:13<2:20:30, 10.15s/it]
 72%|███████▏  | 2154/2984 [6:10:24<2:20:20, 10.14s/it]
 72%|███████▏  | 2155/2984 [6:10:34<2:20:13, 10.15s/it
0: {'loss': 0.5116, 'grad_norm': 0.9120537336869883, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.43}
0: ]
 72%|███████▏  | 2156/2984 [6:10:44<2:20:10, 10.16s/it]
 72%|███████▏  | 2157/2984 [6:10:54<2:19:53, 10.15s/it]
 72%|███████▏  | 2158/2984 [6:11:04<2:19:49, 10.16s/it]
 72%|███████▏  | 2159/2984 [6:11:14<2:19:49, 10.17s/it]
 72%|███████▏  | 2160/2984 [6:11:25<2:20:00, 10.19s/it]
                                                       

 72%|███████▏  | 2160/2984 [6:11:25<2:20:00, 10.19s/it]
 72%|███████▏  | 2161/2984 [6:11:35<2:19:55, 10.20s/it]
 72%|███████▏  | 2162/2984 [6:11:45<2:19:08, 10.16s/it]
 72%|███████▏  | 2163/2984 [6:11:55<2:18:52, 10.15s/it]
 73%|███████▎  | 2164/2984 [6:12:05<2:19:13, 10.19s/it]
 73%|███████▎  | 2165/2984 [6:12:16<2:19:18, 10.21s/it]
 73%|███████▎  | 2166/2984 [6:12:26<2:18:53, 10.19s/it]
 73%|███████▎  | 2167/2984 [6:12:36<2:18:12, 10.15s/it]
 73%|███████▎
0: {'loss': 0.507, 'grad_norm': 0.9112840482843516, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.44}
0:   | 2168/2984 [6:12:46<2:17:28, 10.11s/it]
 73%|███████▎  | 2169/2984 [6:12:56<2:17:04, 10.09s/it]
 73%|███████▎  | 2170/2984 [6:13:06<2:16:48, 10.08s/it]
                                                       

 73%|███████▎  | 2170/2984 [6:13:06<2:16:48, 10.08s/it]
 73%|███████▎  | 2171/2984 [6:13:16<2:16:29, 10.07s/it]
 73%|███████▎  | 2172/2984 [6:13:26<2:16:27, 10.08s/it]
 73%|███████▎  | 2173/2984 [6:13:38<2:21:58, 10.50s/it]
 73%|███████▎  | 2174/2984 [6:13:48<2:19:51, 10.36s/it]
 73%|███████▎  | 2175/2984 [6:13:58<2:19:54, 10.38s/it]
 73%|███████▎  | 2176/2984 [6:14:08<2:19:25, 10.35s/it]
 73%|███████▎  | 2177/2984 [6:14:20<2:22:26, 10.59s/it]
 73%|███████▎  | 2178/2984 [6:14:29<2:19:44, 10.40s/it]
 73%|███████▎  | 2179/2984 [6:14:39<2:17:28, 10.25s/it]
 73%|███████▎  | 2180/2984 [6:14:50<2:19:04,
0: {'loss': 0.5235, 'grad_norm': 0.8885361943775842, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.44}
0: {'loss': 0.5144, 'grad_norm': 0.8901495064067741, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.44}
0:  10.38s/it]
                                                       

 73%|███████▎  | 2180/2984 [6:14:50<2:19:04, 10.38s/it]
 73%|███████▎  | 2181/2984 [6:15:00<2:17:13, 10.25s/it]
 73%|███████▎  | 2182/2984 [6:15:10<2:15:52, 10.16s/it]
 73%|███████▎  | 2183/2984 [6:15:20<2:15:00, 10.11s/it]
 73%|███████▎  | 2184/2984 [6:15:30<2:14:28, 10.09s/it]
 73%|███████▎  | 2185/2984 [6:15:40<2:15:51, 10.20s/it]
 73%|███████▎  | 2186/2984 [6:15:51<2:15:06, 10.16s/it]
 73%|███████▎  | 2187/2984 [6:16:01<2:14:25, 10.12s/it]
 73%|███████▎  | 2188/2984 [6:16:11<2:13:44, 10.08s/it]
 73%|███████▎  | 2189/2984 [6:16:21<2:13:10, 10.05s/it]
 73%|███████▎  | 2190/2984 [6:16:31<2:12:47, 10.03s/it]
                                                       

 73%|███████▎  | 2190/2984 [6:16:31<2:12:47, 10.03s/it]
 73%|███████▎  | 2
0: {'loss': 0.4962, 'grad_norm': 1.0274412322670123, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.44}
0: 191/2984 [6:16:41<2:12:33, 10.03s/it]
 73%|███████▎  | 2192/2984 [6:16:51<2:12:22, 10.03s/it]
 73%|███████▎  | 2193/2984 [6:17:01<2:12:23, 10.04s/it]
 74%|███████▎  | 2194/2984 [6:17:11<2:13:01, 10.10s/it]
 74%|███████▎  | 2195/2984 [6:17:21<2:13:11, 10.13s/it]
 74%|███████▎  | 2196/2984 [6:17:31<2:13:30, 10.17s/it]
 74%|███████▎  | 2197/2984 [6:17:42<2:15:08, 10.30s/it]
 74%|███████▎  | 2198/2984 [6:17:55<2:25:09, 11.08s/it]
 74%|███████▎  | 2199/2984 [6:18:06<2:23:51, 11.00s/it]
 74%|███████▎  | 2200/2984 [6:18:16<2:21:41, 10.84s/it]
                                                       

 74%|███████▎  | 2200/2984 [6:18:16<2:21:41, 10.84s/it]
 74%|███████▍  | 2201/2984 [6:18:27<2:20:41, 10.78s/it]
 74%|███████▍  | 2202/2984 [6:18:37<2:19:25, 10.70s/it]
 74%|███████▍  | 2203/2984 [6:18:48<2:18:18, 10.6
0: {'loss': 0.5207, 'grad_norm': 0.8728978275884248, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.44}
0: 3s/it]
 74%|███████▍  | 2204/2984 [6:18:58<2:17:31, 10.58s/it]
 74%|███████▍  | 2205/2984 [6:19:09<2:16:53, 10.54s/it]
 74%|███████▍  | 2206/2984 [6:19:19<2:16:20, 10.52s/it]
 74%|███████▍  | 2207/2984 [6:19:30<2:15:59, 10.50s/it]
 74%|███████▍  | 2208/2984 [6:19:40<2:15:47, 10.50s/it]
 74%|███████▍  | 2209/2984 [6:19:50<2:15:21, 10.48s/it]
 74%|███████▍  | 2210/2984 [6:20:01<2:14:56, 10.46s/it]
                                                       

 74%|███████▍  | 2210/2984 [6:20:01<2:14:56, 10.46s/it]
 74%|███████▍  | 2211/2984 [6:20:11<2:13:56, 10.40s/it]
 74%|███████▍  | 2212/2984 [6:20:21<2:13:02, 10.34s/it]
 74%|███████▍  | 2213/2984 [6:20:32<2:12:30, 10.31s/it]
 74%|███████▍  | 2214/2984 [6:20:42<2:12:02, 10.29s/it]
 74%|███████▍  | 2215/2984 [6:20:52<2:12:03, 10.30s/it]
 74%|██████�
0: {'loss': 0.5201, 'grad_norm': 0.8504960780933435, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.45}
0: ��▍  | 2216/2984 [6:21:03<2:12:04, 10.32s/it]
 74%|███████▍  | 2217/2984 [6:21:13<2:12:04, 10.33s/it]
 74%|███████▍  | 2218/2984 [6:21:23<2:12:15, 10.36s/it]
 74%|███████▍  | 2219/2984 [6:21:34<2:12:26, 10.39s/it]
 74%|███████▍  | 2220/2984 [6:21:44<2:12:14, 10.39s/it]
                                                       

 74%|███████▍  | 2220/2984 [6:21:44<2:12:14, 10.39s/it]
 74%|███████▍  | 2221/2984 [6:21:55<2:12:50, 10.45s/it]
 74%|███████▍  | 2222/2984 [6:22:05<2:12:52, 10.46s/it]
 74%|███████▍  | 2223/2984 [6:22:16<2:12:17, 10.43s/it]
 75%|███████▍  | 2224/2984 [6:22:26<2:11:29, 10.38s/it]
 75%|███████▍  | 2225/2984 [6:22:36<2:11:26, 10.39s/it]
 75%|███████▍  | 2226/2984 [6:22:47<2:11:30, 10.41s/it]
 75%|███████▍  | 2227/2984 [6:22:57<2:10:43, 10.36s/it]
 75%|███████▍  | 2228/2984 [6:23:07<2:1
0: {'loss': 0.5005, 'grad_norm': 0.8584425478075771, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.45}
0: 0:44, 10.38s/it]
 75%|███████▍  | 2229/2984 [6:23:18<2:10:54, 10.40s/it]
 75%|███████▍  | 2230/2984 [6:23:28<2:10:07, 10.36s/it]
                                                       

 75%|███████▍  | 2230/2984 [6:23:28<2:10:07, 10.36s/it]
 75%|███████▍  | 2231/2984 [6:23:38<2:09:37, 10.33s/it]
 75%|███████▍  | 2232/2984 [6:23:49<2:09:34, 10.34s/it]
 75%|███████▍  | 2233/2984 [6:23:59<2:08:55, 10.30s/it]
 75%|███████▍  | 2234/2984 [6:24:09<2:08:13, 10.26s/it]
 75%|███████▍  | 2235/2984 [6:24:19<2:07:52, 10.24s/it]
 75%|███████▍  | 2236/2984 [6:24:30<2:07:31, 10.23s/it]
 75%|███████▍  | 2237/2984 [6:24:40<2:08:03, 10.29s/it]
 75%|███████▌  | 2238/2984 [6:24:50<2:08:06, 10.30s/it]
 75%|███████▌  | 2239/2984 [6:25:01<2:07:48, 10.29s/it]
 75%|███████▌  | 2240/2984 [6:25:11<2:07:40, 10.30s/it]
              
0: {'loss': 0.4952, 'grad_norm': 0.9267902579916931, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.45}
0: {'loss': 0.4992, 'grad_norm': 0.873977883524081, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.45}
0:                                          

 75%|███████▌  | 2240/2984 [6:25:11<2:07:40, 10.30s/it]
 75%|███████▌  | 2241/2984 [6:25:21<2:07:28, 10.29s/it]
 75%|███████▌  | 2242/2984 [6:25:32<2:07:39, 10.32s/it]
 75%|███████▌  | 2243/2984 [6:25:42<2:07:50, 10.35s/it]
 75%|███████▌  | 2244/2984 [6:25:52<2:08:01, 10.38s/it]
 75%|███████▌  | 2245/2984 [6:26:03<2:08:08, 10.40s/it]
 75%|███████▌  | 2246/2984 [6:26:13<2:08:18, 10.43s/it]
 75%|███████▌  | 2247/2984 [6:26:24<2:08:07, 10.43s/it]
 75%|███████▌  | 2248/2984 [6:26:34<2:08:24, 10.47s/it]
 75%|███████▌  | 2249/2984 [6:26:45<2:08:24, 10.48s/it]
 75%|███████▌  | 2250/2984 [6:26:55<2:08:45, 10.53s/it]
                                                       

 75%|███████▌  | 2250/2984 [6:26:55<2:08:45, 10.53s/it]
 75%|███████▌  | 2251/2984 [6:27:06<2:09:06,
0: {'loss': 0.5084, 'grad_norm': 0.9095948857093753, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.45}
0:  10.57s/it]
 75%|███████▌  | 2252/2984 [6:27:17<2:08:46, 10.56s/it]
 76%|███████▌  | 2253/2984 [6:27:27<2:08:06, 10.52s/it]
 76%|███████▌  | 2254/2984 [6:27:38<2:07:50, 10.51s/it]
 76%|███████▌  | 2255/2984 [6:27:48<2:07:36, 10.50s/it]
 76%|███████▌  | 2256/2984 [6:27:59<2:07:40, 10.52s/it]
 76%|███████▌  | 2257/2984 [6:28:09<2:07:21, 10.51s/it]
 76%|███████▌  | 2258/2984 [6:28:20<2:07:18, 10.52s/it]
 76%|███████▌  | 2259/2984 [6:28:30<2:07:01, 10.51s/it]
 76%|███████▌  | 2260/2984 [6:28:41<2:06:20, 10.47s/it]
                                                       

 76%|███████▌  | 2260/2984 [6:28:41<2:06:20, 10.47s/it]
 76%|███████▌  | 2261/2984 [6:28:51<2:05:27, 10.41s/it]
 76%|███████▌  | 2262/2984 [6:29:01<2:04:47, 10.37s/it]
 76%|███████▌  | 2263/2984 [6:29:11<2:04:14, 10.34s/it]
 76%|████�
0: {'loss': 0.4993, 'grad_norm': 0.9444298712179964, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.46}
0: �██▌  | 2264/2984 [6:29:22<2:03:58, 10.33s/it]
 76%|███████▌  | 2265/2984 [6:29:32<2:03:36, 10.31s/it]
 76%|███████▌  | 2266/2984 [6:29:42<2:03:08, 10.29s/it]
 76%|███████▌  | 2267/2984 [6:29:52<2:02:46, 10.27s/it]
 76%|███████▌  | 2268/2984 [6:30:03<2:02:30, 10.27s/it]
 76%|███████▌  | 2269/2984 [6:30:13<2:03:03, 10.33s/it]
 76%|███████▌  | 2270/2984 [6:30:24<2:03:13, 10.36s/it]
                                                       

 76%|███████▌  | 2270/2984 [6:30:24<2:03:13, 10.36s/it]
 76%|███████▌  | 2271/2984 [6:30:34<2:02:46, 10.33s/it]
 76%|███████▌  | 2272/2984 [6:30:44<2:03:08, 10.38s/it]
 76%|███████▌  | 2273/2984 [6:30:55<2:03:12, 10.40s/it]
 76%|███████▌  | 2274/2984 [6:31:05<2:02:49, 10.38s/it]
 76%|███████▌  | 2275/2984 [6:31:15<2:02:29, 10.37s/it]
 76%|███████▋  | 2276/2984 [6:31:2
0: {'loss': 0.5064, 'grad_norm': 0.9296672881438112, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.46}
0: 6<2:02:18, 10.36s/it]
 76%|███████▋  | 2277/2984 [6:31:36<2:02:02, 10.36s/it]
 76%|███████▋  | 2278/2984 [6:31:47<2:01:59, 10.37s/it]
 76%|███████▋  | 2279/2984 [6:31:57<2:01:34, 10.35s/it]
 76%|███████▋  | 2280/2984 [6:32:07<2:02:12, 10.42s/it]
                                                       

 76%|███████▋  | 2280/2984 [6:32:07<2:02:12, 10.42s/it]
 76%|███████▋  | 2281/2984 [6:32:18<2:02:24, 10.45s/it]
 76%|███████▋  | 2282/2984 [6:32:28<2:02:25, 10.46s/it]
 77%|███████▋  | 2283/2984 [6:32:39<2:01:50, 10.43s/it]
 77%|███████▋  | 2284/2984 [6:32:49<2:01:22, 10.40s/it]
 77%|███████▋  | 2285/2984 [6:32:59<2:01:02, 10.39s/it]
 77%|███████▋  | 2286/2984 [6:33:10<2:00:47, 10.38s/it]
 77%|███████▋  | 2287/2984 [6:33:20<2:00:15, 10.35s/it]
 77%|███████▋  | 2288/2984 [6:33:30<1:59:56, 10.34s/it]
 77%|█�
0: {'loss': 0.5043, 'grad_norm': 0.919985936695404, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.46}
0: ��█████▋  | 2289/2984 [6:33:41<1:59:21, 10.30s/it]
 77%|███████▋  | 2290/2984 [6:33:51<1:59:15, 10.31s/it]
                                                       

 77%|███████▋  | 2290/2984 [6:33:51<1:59:15, 10.31s/it]
 77%|███████▋  | 2291/2984 [6:34:01<1:59:04, 10.31s/it]
 77%|███████▋  | 2292/2984 [6:34:12<1:58:41, 10.29s/it]
 77%|███████▋  | 2293/2984 [6:34:22<1:58:40, 10.31s/it]
 77%|███████▋  | 2294/2984 [6:34:32<1:58:48, 10.33s/it]
 77%|███████▋  | 2295/2984 [6:34:43<1:58:51, 10.35s/it]
 77%|███████▋  | 2296/2984 [6:34:53<1:58:37, 10.34s/it]
 77%|███████▋  | 2297/2984 [6:35:03<1:58:19, 10.33s/it]
 77%|███████▋  | 2298/2984 [6:35:14<1:58:00, 10.32s/it]
 77%|███████▋  | 2299/2984 [6:35:24<1:57:52, 10.33s/it]
 77%|███████▋  | 2300/2984 [6:35:34<1:57:40, 10.32s/it]
                                        
0: {'loss': 0.5068, 'grad_norm': 0.9243829460306879, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.46}
0: {'loss': 0.497, 'grad_norm': 0.8454886150085394, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.46}
0:                

 77%|███████▋  | 2300/2984 [6:35:34<1:57:40, 10.32s/it]
 77%|███████▋  | 2301/2984 [6:35:44<1:57:16, 10.30s/it]
 77%|███████▋  | 2302/2984 [6:35:55<1:56:47, 10.28s/it]
 77%|███████▋  | 2303/2984 [6:36:05<1:56:26, 10.26s/it]
 77%|███████▋  | 2304/2984 [6:36:15<1:56:21, 10.27s/it]
 77%|███████▋  | 2305/2984 [6:36:26<1:56:17, 10.28s/it]
 77%|███████▋  | 2306/2984 [6:36:36<1:57:29, 10.40s/it]
 77%|███████▋  | 2307/2984 [6:36:47<1:57:57, 10.45s/it]
 77%|███████▋  | 2308/2984 [6:36:57<1:57:12, 10.40s/it]
 77%|███████▋  | 2309/2984 [6:37:07<1:56:25, 10.35s/it]
 77%|███████▋  | 2310/2984 [6:37:17<1:55:27, 10.28s/it]
                                                       

 77%|███████▋  | 2310/2984 [6:37:17<1:55:27, 10.28s/it]
 77%|███████▋  | 2311/2984 [6:37:27<1:54:31, 10.21s/it]
 77%|███
0: {'loss': 0.5028, 'grad_norm': 0.9589048060662106, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.47}
0: ████▋  | 2312/2984 [6:37:38<1:53:57, 10.17s/it]
 78%|███████▊  | 2313/2984 [6:37:48<1:53:25, 10.14s/it]
 78%|███████▊  | 2314/2984 [6:37:58<1:53:00, 10.12s/it]
 78%|███████▊  | 2315/2984 [6:38:08<1:53:24, 10.17s/it]
 78%|███████▊  | 2316/2984 [6:38:18<1:52:51, 10.14s/it]
 78%|███████▊  | 2317/2984 [6:38:28<1:52:21, 10.11s/it]
 78%|███████▊  | 2318/2984 [6:38:38<1:52:02, 10.09s/it]
 78%|███████▊  | 2319/2984 [6:38:48<1:51:55, 10.10s/it]
 78%|███████▊  | 2320/2984 [6:38:58<1:52:06, 10.13s/it]
                                                       

 78%|███████▊  | 2320/2984 [6:38:58<1:52:06, 10.13s/it]
 78%|███████▊  | 2321/2984 [6:39:09<1:52:26, 10.18s/it]
 78%|███████▊  | 2322/2984 [6:39:19<1:52:16, 10.18s/it]
 78%|███████▊  | 2323/2984 [6:39:29<1:52:11, 10.18s/it]
 78%|███████▊  | 2324/2984 [6
0: {'loss': 0.5034, 'grad_norm': 0.8713306668545948, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.47}
0: :39:39<1:52:20, 10.21s/it]
 78%|███████▊  | 2325/2984 [6:39:50<1:52:16, 10.22s/it]
 78%|███████▊  | 2326/2984 [6:40:00<1:52:17, 10.24s/it]
 78%|███████▊  | 2327/2984 [6:40:10<1:52:06, 10.24s/it]
 78%|███████▊  | 2328/2984 [6:40:20<1:51:53, 10.23s/it]
 78%|███████▊  | 2329/2984 [6:40:31<1:51:51, 10.25s/it]
 78%|███████▊  | 2330/2984 [6:40:41<1:51:49, 10.26s/it]
                                                       

 78%|███████▊  | 2330/2984 [6:40:41<1:51:49, 10.26s/it]
 78%|███████▊  | 2331/2984 [6:40:51<1:52:00, 10.29s/it]
 78%|███████▊  | 2332/2984 [6:41:02<1:51:57, 10.30s/it]
 78%|███████▊  | 2333/2984 [6:41:12<1:51:50, 10.31s/it]
 78%|███████▊  | 2334/2984 [6:41:22<1:52:01, 10.34s/it]
 78%|███████▊  | 2335/2984 [6:41:33<1:52:05, 10.36s/it]
 78%|███████▊  | 2336/2984 [6:41:43<1:51:55, 10.36s/it]
 78%
0: {'loss': 0.5039, 'grad_norm': 0.9446977082218846, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.47}
0: |███████▊  | 2337/2984 [6:41:54<1:51:52, 10.37s/it]
 78%|███████▊  | 2338/2984 [6:42:04<1:51:34, 10.36s/it]
 78%|███████▊  | 2339/2984 [6:42:14<1:51:24, 10.36s/it]
 78%|███████▊  | 2340/2984 [6:42:25<1:51:09, 10.36s/it]
                                                       

 78%|███████▊  | 2340/2984 [6:42:25<1:51:09, 10.36s/it]
 78%|███████▊  | 2341/2984 [6:42:35<1:51:13, 10.38s/it]
 78%|███████▊  | 2342/2984 [6:42:45<1:50:54, 10.37s/it]
 79%|███████▊  | 2343/2984 [6:42:56<1:50:26, 10.34s/it]
 79%|███████▊  | 2344/2984 [6:43:06<1:50:08, 10.33s/it]
 79%|███████▊  | 2345/2984 [6:43:16<1:49:58, 10.33s/it]
 79%|███████▊  | 2346/2984 [6:43:26<1:49:35, 10.31s/it]
 79%|███████▊  | 2347/2984 [6:43:37<1:48:59, 10.27s/it]
 79%|███████▊  | 2348/2984 [6:43:47<1:48:23, 10.23s/it]
 79%|███████▊  | 23
0: {'loss': 0.4912, 'grad_norm': 0.8608509974389424, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.47}
0: {'loss': 0.509, 'grad_norm': 0.9053553777135821, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.47}
0: 49/2984 [6:43:57<1:47:59, 10.20s/it]
 79%|███████▉  | 2350/2984 [6:44:07<1:47:36, 10.18s/it]
                                                       

 79%|███████▉  | 2350/2984 [6:44:07<1:47:36, 10.18s/it]
 79%|███████▉  | 2351/2984 [6:44:17<1:47:25, 10.18s/it]
 79%|███████▉  | 2352/2984 [6:44:27<1:47:03, 10.16s/it]
 79%|███████▉  | 2353/2984 [6:44:37<1:46:43, 10.15s/it]
 79%|███████▉  | 2354/2984 [6:44:48<1:46:24, 10.13s/it]
 79%|███████▉  | 2355/2984 [6:44:58<1:46:03, 10.12s/it]
 79%|███████▉  | 2356/2984 [6:45:08<1:46:00, 10.13s/it]
 79%|███████▉  | 2357/2984 [6:45:18<1:45:57, 10.14s/it]
 79%|███████▉  | 2358/2984 [6:45:28<1:45:57, 10.16s/it]
 79%|███████▉  | 2359/2984 [6:45:38<1:46:00, 10.18s/it]
 79%|███████▉  | 2360/2984 [6:45:49<1:46:04, 10.20s/it]
                                                       

 79%|█�
0: {'loss': 0.5064, 'grad_norm': 1.0172871238825527, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.48}
0: ��█████▉  | 2360/2984 [6:45:49<1:46:04, 10.20s/it]
 79%|███████▉  | 2361/2984 [6:45:59<1:46:53, 10.29s/it]
 79%|███████▉  | 2362/2984 [6:46:09<1:46:29, 10.27s/it]
 79%|███████▉  | 2363/2984 [6:46:20<1:46:01, 10.24s/it]
 79%|███████▉  | 2364/2984 [6:46:30<1:45:41, 10.23s/it]
 79%|███████▉  | 2365/2984 [6:46:41<1:48:27, 10.51s/it]
 79%|███████▉  | 2366/2984 [6:46:52<1:49:15, 10.61s/it]
 79%|███████▉  | 2367/2984 [6:47:02<1:47:44, 10.48s/it]
 79%|███████▉  | 2368/2984 [6:47:12<1:46:34, 10.38s/it]
 79%|███████▉  | 2369/2984 [6:47:22<1:45:49, 10.32s/it]
 79%|███████▉  | 2370/2984 [6:47:33<1:45:20, 10.29s/it]
                                                       

 79%|███████▉  | 2370/2984 [6:47:33<1:45:20, 10.29s/it]
 79%|███████▉  | 2371/2984 [6:47:43<1:45:24, 10.32s/it]
 79%|███████▉  | 2372/29
0: {'loss': 0.4991, 'grad_norm': 0.9076211341117741, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.48}
0: 84 [6:47:53<1:45:14, 10.32s/it]
 80%|███████▉  | 2373/2984 [6:48:03<1:44:57, 10.31s/it]
 80%|███████▉  | 2374/2984 [6:48:14<1:44:46, 10.31s/it]
 80%|███████▉  | 2375/2984 [6:48:24<1:44:35, 10.30s/it]
 80%|███████▉  | 2376/2984 [6:48:34<1:44:16, 10.29s/it]
 80%|███████▉  | 2377/2984 [6:48:45<1:44:09, 10.30s/it]
 80%|███████▉  | 2378/2984 [6:48:55<1:43:42, 10.27s/it]
 80%|███████▉  | 2379/2984 [6:49:05<1:44:08, 10.33s/it]
 80%|███████▉  | 2380/2984 [6:49:16<1:44:20, 10.37s/it]
                                                       

 80%|███████▉  | 2380/2984 [6:49:16<1:44:20, 10.37s/it]
 80%|███████▉  | 2381/2984 [6:49:26<1:44:26, 10.39s/it]
 80%|███████▉  | 2382/2984 [6:49:37<1:44:24, 10.41s/it]
 80%|███████▉  | 2383/2984 [6:49:47<1:44:22, 10.42s/it]
 80%|███████▉  | 2384/2984 [6:49:58<1:44:15, 10.43s/it]
0: {'loss': 0.5035, 'grad_norm': 1.050318094680644, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.48}
0: 
 80%|███████▉  | 2385/2984 [6:50:08<1:44:10, 10.44s/it]
 80%|███████▉  | 2386/2984 [6:50:18<1:44:04, 10.44s/it]
 80%|███████▉  | 2387/2984 [6:50:29<1:44:23, 10.49s/it]
 80%|████████  | 2388/2984 [6:50:40<1:44:31, 10.52s/it]
 80%|████████  | 2389/2984 [6:50:50<1:44:42, 10.56s/it]
 80%|████████  | 2390/2984 [6:51:01<1:44:48, 10.59s/it]
                                                       

 80%|████████  | 2390/2984 [6:51:01<1:44:48, 10.59s/it]
 80%|████████  | 2391/2984 [6:51:12<1:44:25, 10.57s/it]
 80%|████████  | 2392/2984 [6:51:22<1:44:03, 10.55s/it]
 80%|████████  | 2393/2984 [6:51:32<1:43:38, 10.52s/it]
 80%|████████  | 2394/2984 [6:51:43<1:44:03, 10.58s/it]
 80%|████████  | 2395/2984 [6:51:54<1:43:25, 10.54s/it]
 80%|████████  | 2396/2984 [6:52:04<1:42:48, 10.49s/it]
 80%|████████ 
0: {'loss': 0.499, 'grad_norm': 0.8578201926168108, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.48}
0:  | 2397/2984 [6:52:14<1:42:28, 10.48s/it]
 80%|████████  | 2398/2984 [6:52:25<1:42:05, 10.45s/it]
 80%|████████  | 2399/2984 [6:52:35<1:41:22, 10.40s/it]
 80%|████████  | 2400/2984 [6:52:45<1:40:42, 10.35s/it]
                                                       

 80%|████████  | 2400/2984 [6:52:45<1:40:42, 10.35s/it]
 80%|████████  | 2401/2984 [6:52:56<1:40:16, 10.32s/it]
 80%|████████  | 2402/2984 [6:53:06<1:39:47, 10.29s/it]
 81%|████████  | 2403/2984 [6:53:16<1:39:22, 10.26s/it]
 81%|████████  | 2404/2984 [6:53:26<1:38:59, 10.24s/it]
 81%|████████  | 2405/2984 [6:53:36<1:38:48, 10.24s/it]
 81%|████████  | 2406/2984 [6:53:47<1:38:41, 10.25s/it]
 81%|████████  | 2407/2984 [6:53:57<1:38:35, 10.25s/it]
 81%|████████  | 2408/2984 [6:54:07<1:38:31, 10.26s/it]
 81%|████████  | 2409/2984 [6:54:18<1:38:32, 
0: {'loss': 0.5016, 'grad_norm': 0.8885896025671491, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.48}
0: {'loss': 0.502, 'grad_norm': 0.9121407066800825, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.49}
0: 10.28s/it]
 81%|████████  | 2410/2984 [6:54:28<1:38:30, 10.30s/it]
                                                       

 81%|████████  | 2410/2984 [6:54:28<1:38:30, 10.30s/it]
 81%|████████  | 2411/2984 [6:54:38<1:38:19, 10.30s/it]
 81%|████████  | 2412/2984 [6:54:48<1:38:02, 10.28s/it]
 81%|████████  | 2413/2984 [6:54:59<1:38:00, 10.30s/it]
 81%|████████  | 2414/2984 [6:55:09<1:37:59, 10.32s/it]
 81%|████████  | 2415/2984 [6:55:20<1:38:10, 10.35s/it]
 81%|████████  | 2416/2984 [6:55:30<1:38:05, 10.36s/it]
 81%|████████  | 2417/2984 [6:55:40<1:38:08, 10.39s/it]
 81%|████████  | 2418/2984 [6:55:51<1:39:08, 10.51s/it]
 81%|████████  | 2419/2984 [6:56:02<1:38:54, 10.50s/it]
 81%|████████  | 2420/2984 [6:56:12<1:38:37, 10.49s/it]
                                                       

 81%|████████  | 24
0: {'loss': 0.5044, 'grad_norm': 0.8618126736229195, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.49}
0: 20/2984 [6:56:12<1:38:37, 10.49s/it]
 81%|████████  | 2421/2984 [6:56:23<1:38:37, 10.51s/it]
 81%|████████  | 2422/2984 [6:56:33<1:38:19, 10.50s/it]
 81%|████████  | 2423/2984 [6:56:44<1:38:06, 10.49s/it]
 81%|████████  | 2424/2984 [6:56:54<1:38:03, 10.51s/it]
 81%|████████▏ | 2425/2984 [6:57:05<1:37:27, 10.46s/it]
 81%|████████▏ | 2426/2984 [6:57:15<1:36:48, 10.41s/it]
 81%|████████▏ | 2427/2984 [6:57:25<1:36:18, 10.37s/it]
 81%|████████▏ | 2428/2984 [6:57:36<1:38:20, 10.61s/it]
 81%|████████▏ | 2429/2984 [6:57:47<1:38:09, 10.61s/it]
 81%|████████▏ | 2430/2984 [6:57:57<1:37:25, 10.55s/it]
                                                       

 81%|████████▏ | 2430/2984 [6:57:57<1:37:25, 10.55s/it]
 81%|████████▏ | 2431/2984 [6:58:08<1:36:59, 10.52s/it]
 82%|████████▏ | 2432/2984 [6:58
0: {'loss': 0.5036, 'grad_norm': 0.7752225035416509, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.49}
0: :18<1:36:12, 10.46s/it]
 82%|████████▏ | 2433/2984 [6:58:28<1:35:27, 10.39s/it]
 82%|████████▏ | 2434/2984 [6:58:39<1:34:56, 10.36s/it]
 82%|████████▏ | 2435/2984 [6:58:49<1:34:36, 10.34s/it]
 82%|████████▏ | 2436/2984 [6:58:59<1:34:20, 10.33s/it]
 82%|████████▏ | 2437/2984 [6:59:10<1:34:07, 10.33s/it]
 82%|████████▏ | 2438/2984 [6:59:20<1:33:54, 10.32s/it]
 82%|████████▏ | 2439/2984 [6:59:30<1:33:49, 10.33s/it]
 82%|████████▏ | 2440/2984 [6:59:41<1:33:44, 10.34s/it]
                                                       

 82%|████████▏ | 2440/2984 [6:59:41<1:33:44, 10.34s/it]
 82%|████████▏ | 2441/2984 [6:59:51<1:33:39, 10.35s/it]
 82%|████████▏ | 2442/2984 [7:00:01<1:33:10, 10.32s/it]
 82%|████████▏ | 2443/2984 [7:00:11<1:32:32, 10.26s/it]
 82%|████████▏ | 2444/2984 [7:00:21<1
0: {'loss': 0.5048, 'grad_norm': 0.8532534677001996, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.49}
0: :32:06, 10.23s/it]
 82%|████████▏ | 2445/2984 [7:00:32<1:31:35, 10.20s/it]
 82%|████████▏ | 2446/2984 [7:00:42<1:31:03, 10.16s/it]
 82%|████████▏ | 2447/2984 [7:00:52<1:30:58, 10.17s/it]
 82%|████████▏ | 2448/2984 [7:01:02<1:31:03, 10.19s/it]
 82%|████████▏ | 2449/2984 [7:01:12<1:31:12, 10.23s/it]
 82%|████████▏ | 2450/2984 [7:01:23<1:31:12, 10.25s/it]
                                                       

 82%|████████▏ | 2450/2984 [7:01:23<1:31:12, 10.25s/it]
 82%|████████▏ | 2451/2984 [7:01:33<1:31:13, 10.27s/it]
 82%|████████▏ | 2452/2984 [7:01:43<1:31:13, 10.29s/it]
 82%|████████▏ | 2453/2984 [7:01:54<1:31:17, 10.32s/it]
 82%|████████▏ | 2454/2984 [7:02:04<1:31:10, 10.32s/it]
 82%|████████▏ | 2455/2984 [7:02:14<1:30:55, 10.31s/it]
 82%|████████▏ | 2456/2984 [7:02:25<1:30:4
0: {'loss': 0.4962, 'grad_norm': 0.8626520667500694, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.49}
0: 5, 10.31s/it]
 82%|████████▏ | 2457/2984 [7:02:35<1:30:47, 10.34s/it]
 82%|████████▏ | 2458/2984 [7:02:45<1:30:30, 10.32s/it]
 82%|████████▏ | 2459/2984 [7:02:56<1:30:13, 10.31s/it]
 82%|████████▏ | 2460/2984 [7:03:06<1:30:02, 10.31s/it]
                                                       

 82%|████████▏ | 2460/2984 [7:03:06<1:30:02, 10.31s/it]
 82%|████████▏ | 2461/2984 [7:03:16<1:29:53, 10.31s/it]
 83%|████████▎ | 2462/2984 [7:03:27<1:29:34, 10.30s/it]
 83%|████████▎ | 2463/2984 [7:03:37<1:29:10, 10.27s/it]
 83%|████████▎ | 2464/2984 [7:03:47<1:28:43, 10.24s/it]
 83%|████████▎ | 2465/2984 [7:03:57<1:28:30, 10.23s/it]
 83%|████████▎ | 2466/2984 [7:04:07<1:28:20, 10.23s/it]
 83%|████████▎ | 2467/2984 [7:04:18<1:28:15, 10.24s/it]
 83%|████████▎ | 2468/2984 [7:04:28<1:28:07, 10
0: {'loss': 0.4923, 'grad_norm': 0.8882675610994514, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.5}
0: .25s/it]
 83%|████████▎ | 2469/2984 [7:04:38<1:28:04, 10.26s/it]
 83%|████████▎ | 2470/2984 [7:04:48<1:27:56, 10.26s/it]
                                                       

 83%|████████▎ | 2470/2984 [7:04:48<1:27:56, 10.26s/it]
 83%|████████▎ | 2471/2984 [7:04:59<1:27:48, 10.27s/it]
 83%|████████▎ | 2472/2984 [7:05:09<1:27:50, 10.29s/it]
 83%|████████▎ | 2473/2984 [7:05:19<1:27:49, 10.31s/it]
 83%|████████▎ | 2474/2984 [7:05:30<1:27:47, 10.33s/it]
 83%|████████▎ | 2475/2984 [7:05:40<1:27:42, 10.34s/it]
 83%|████████▎ | 2476/2984 [7:05:51<1:27:34, 10.34s/it]
 83%|████████▎ | 2477/2984 [7:06:01<1:27:28, 10.35s/it]
 83%|████████▎ | 2478/2984 [7:06:11<1:27:13, 10.34s/it]
 83%|████████▎ | 2479/2984 [7:06:22<1:27:02, 10.34s/it]
 83%|████████▎ | 2480/2984 [7:06:32<1:26:46, 10.33s/
0: {'loss': 0.4972, 'grad_norm': 0.9058703590982193, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.5}
0: {'loss': 0.5073, 'grad_norm': 0.8685711948160576, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.5}
0: it]
                                                       

 83%|████████▎ | 2480/2984 [7:06:32<1:26:46, 10.33s/it]
 83%|████████▎ | 2481/2984 [7:06:42<1:26:39, 10.34s/it]
 83%|████████▎ | 2482/2984 [7:06:53<1:26:34, 10.35s/it]
 83%|████████▎ | 2483/2984 [7:07:03<1:26:29, 10.36s/it]
 83%|████████▎ | 2484/2984 [7:07:13<1:26:19, 10.36s/it]
 83%|████████▎ | 2485/2984 [7:07:24<1:26:18, 10.38s/it]
 83%|████████▎ | 2486/2984 [7:07:34<1:26:17, 10.40s/it]
 83%|████████▎ | 2487/2984 [7:07:45<1:26:33, 10.45s/it]
 83%|████████▎ | 2488/2984 [7:07:55<1:26:30, 10.46s/it]
 83%|████████▎ | 2489/2984 [7:08:06<1:26:14, 10.45s/it]
 83%|████████▎ | 2490/2984 [7:08:16<1:26:04, 10.45s/it]
                                                       

 83%|████████▎ | 2490/2984 [7:08:16<1:26:04, 10.45s/it]
 83%|████�
0: {'loss': 0.4911, 'grad_norm': 0.8352598310954639, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.5}
0: ��███▎ | 2491/2984 [7:08:27<1:26:03, 10.47s/it]
 84%|████████▎ | 2492/2984 [7:08:37<1:25:40, 10.45s/it]
 84%|████████▎ | 2493/2984 [7:08:47<1:25:26, 10.44s/it]
 84%|████████▎ | 2494/2984 [7:08:58<1:24:57, 10.40s/it]
 84%|████████▎ | 2495/2984 [7:09:08<1:24:44, 10.40s/it]
 84%|████████▎ | 2496/2984 [7:09:19<1:24:25, 10.38s/it]
 84%|████████▎ | 2497/2984 [7:09:29<1:24:03, 10.36s/it]
 84%|████████▎ | 2498/2984 [7:09:39<1:23:46, 10.34s/it]
 84%|████████▎ | 2499/2984 [7:09:49<1:23:34, 10.34s/it]
 84%|████████▍ | 2500/2984 [7:10:00<1:23:20, 10.33s/it]
                                                       

 84%|████████▍ | 2500/2984 [7:10:00<1:23:20, 10.33s/it]
 84%|████████▍ | 2501/2984 [7:10:10<1:23:17, 10.35s/it]
 84%|████████▍ | 2502/2984 [7:10:21<1:23:26, 10.39s/it]
 84%|██████
0: {'loss': 0.5026, 'grad_norm': 0.9665028760431313, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.5}
0: ██▍ | 2503/2984 [7:10:31<1:23:20, 10.40s/it]
 84%|████████▍ | 2504/2984 [7:10:41<1:23:06, 10.39s/it]
 84%|████████▍ | 2505/2984 [7:10:52<1:22:49, 10.38s/it]
 84%|████████▍ | 2506/2984 [7:11:02<1:22:34, 10.36s/it]
 84%|████████▍ | 2507/2984 [7:11:12<1:21:53, 10.30s/it]
 84%|████████▍ | 2508/2984 [7:11:22<1:21:20, 10.25s/it]
 84%|████████▍ | 2509/2984 [7:11:33<1:20:57, 10.23s/it]
 84%|████████▍ | 2510/2984 [7:11:43<1:20:56, 10.25s/it]
                                                       

 84%|████████▍ | 2510/2984 [7:11:43<1:20:56, 10.25s/it]
 84%|████████▍ | 2511/2984 [7:11:53<1:21:00, 10.28s/it]
 84%|████████▍ | 2512/2984 [7:12:04<1:21:06, 10.31s/it]
 84%|████████▍ | 2513/2984 [7:12:14<1:21:05, 10.33s/it]
 84%|████████▍ | 2514/2984 [7:12:24<1:21:11, 10.36s/it]
 84%|███████�
0: {'loss': 0.5019, 'grad_norm': 0.9366506341116507, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.51}
0: �▍ | 2515/2984 [7:12:35<1:20:57, 10.36s/it]
 84%|████████▍ | 2516/2984 [7:12:45<1:20:47, 10.36s/it]
 84%|████████▍ | 2517/2984 [7:12:56<1:20:47, 10.38s/it]
 84%|████████▍ | 2518/2984 [7:13:06<1:20:40, 10.39s/it]
 84%|████████▍ | 2519/2984 [7:13:16<1:20:27, 10.38s/it]
 84%|████████▍ | 2520/2984 [7:13:26<1:19:48, 10.32s/it]
                                                       

 84%|████████▍ | 2520/2984 [7:13:27<1:19:48, 10.32s/it]
 84%|████████▍ | 2521/2984 [7:13:37<1:19:25, 10.29s/it]
 85%|████████▍ | 2522/2984 [7:13:47<1:19:02, 10.26s/it]
 85%|████████▍ | 2523/2984 [7:13:57<1:18:28, 10.21s/it]
 85%|████████▍ | 2524/2984 [7:14:07<1:18:09, 10.19s/it]
 85%|████████▍ | 2525/2984 [7:14:17<1:17:46, 10.17s/it]
 85%|████████▍ | 2526/2984 [7:14:27<1:17:38, 10.17s/it]
 85%|████████▍ 
0: {'loss': 0.4911, 'grad_norm': 1.0477444020199949, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.51}
0: | 2527/2984 [7:14:38<1:17:49, 10.22s/it]
 85%|████████▍ | 2528/2984 [7:14:48<1:17:54, 10.25s/it]
 85%|████████▍ | 2529/2984 [7:14:58<1:17:44, 10.25s/it]
 85%|████████▍ | 2530/2984 [7:15:09<1:17:54, 10.30s/it]
                                                       

 85%|████████▍ | 2530/2984 [7:15:09<1:17:54, 10.30s/it]
 85%|████████▍ | 2531/2984 [7:15:19<1:17:50, 10.31s/it]
 85%|████████▍ | 2532/2984 [7:15:29<1:17:34, 10.30s/it]
 85%|████████▍ | 2533/2984 [7:15:40<1:17:21, 10.29s/it]
 85%|████████▍ | 2534/2984 [7:15:50<1:17:16, 10.30s/it]
 85%|████████▍ | 2535/2984 [7:16:00<1:16:58, 10.29s/it]
 85%|████████▍ | 2536/2984 [7:16:10<1:16:34, 10.26s/it]
 85%|████████▌ | 2537/2984 [7:16:21<1:16:17, 10.24s/it]
 85%|████████▌ | 2538/2984 [7:16:31<1:16:11, 10.25s/it]
 85%|████████▌ | 253
0: {'loss': 0.5021, 'grad_norm': 0.8818573703058795, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.51}
0: 9/2984 [7:16:41<1:16:06, 10.26s/it]
 85%|████████▌ | 2540/2984 [7:16:51<1:16:00, 10.27s/it]
                                                       

 85%|████████▌ | 2540/2984 [7:16:51<1:16:00, 10.27s/it]
 85%|████████▌ | 2541/2984 [7:17:02<1:15:46, 10.26s/it]
 85%|████████▌ | 2542/2984 [7:17:12<1:15:59, 10.32s/it]
 85%|████████▌ | 2543/2984 [7:17:22<1:15:48, 10.31s/it]
 85%|████████▌ | 2544/2984 [7:17:33<1:15:31, 10.30s/it]
 85%|████████▌ | 2545/2984 [7:17:43<1:15:36, 10.33s/it]
 85%|████████▌ | 2546/2984 [7:17:54<1:15:50, 10.39s/it]
 85%|████████▌ | 2547/2984 [7:18:04<1:15:31, 10.37s/it]
 85%|████████▌ | 2548/2984 [7:18:14<1:15:03, 10.33s/it]
 85%|████████▌ | 2549/2984 [7:18:24<1:14:45, 10.31s/it]
 85%|████████▌ | 2550/2984 [7:18:35<1:14:20, 10.28s/it]
                                           
0: {'loss': 0.5004, 'grad_norm': 0.8422375484651724, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.51}
0: {'loss': 0.4934, 'grad_norm': 1.040049435153039, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.51}
0:             

 85%|████████▌ | 2550/2984 [7:18:35<1:14:20, 10.28s/it]
 85%|████████▌ | 2551/2984 [7:18:45<1:14:00, 10.26s/it]
 86%|████████▌ | 2552/2984 [7:18:55<1:13:43, 10.24s/it]
 86%|████████▌ | 2553/2984 [7:19:05<1:13:11, 10.19s/it]
 86%|████████▌ | 2554/2984 [7:19:15<1:12:56, 10.18s/it]
 86%|████████▌ | 2555/2984 [7:19:26<1:12:54, 10.20s/it]
 86%|████████▌ | 2556/2984 [7:19:36<1:12:26, 10.16s/it]
 86%|████████▌ | 2557/2984 [7:19:46<1:12:05, 10.13s/it]
 86%|████████▌ | 2558/2984 [7:19:56<1:11:54, 10.13s/it]
 86%|████████▌ | 2559/2984 [7:20:06<1:11:32, 10.10s/it]
 86%|████████▌ | 2560/2984 [7:20:16<1:11:24, 10.11s/it]
                                                       

 86%|████████▌ | 2560/2984 [7:20:16<1:11:24, 10.11s/it]
 86%|████████▌ | 2561/2984 [7:20:26<1:11:20, 10
0: {'loss': 0.4907, 'grad_norm': 0.8919505495399644, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.52}
0: .12s/it]
 86%|████████▌ | 2562/2984 [7:20:36<1:11:21, 10.15s/it]
 86%|████████▌ | 2563/2984 [7:20:47<1:11:17, 10.16s/it]
 86%|████████▌ | 2564/2984 [7:20:57<1:11:09, 10.17s/it]
 86%|████████▌ | 2565/2984 [7:21:07<1:11:04, 10.18s/it]
 86%|████████▌ | 2566/2984 [7:21:17<1:11:00, 10.19s/it]
 86%|████████▌ | 2567/2984 [7:21:27<1:10:52, 10.20s/it]
 86%|████████▌ | 2568/2984 [7:21:38<1:10:47, 10.21s/it]
 86%|████████▌ | 2569/2984 [7:21:48<1:10:32, 10.20s/it]
 86%|████████▌ | 2570/2984 [7:21:58<1:10:11, 10.17s/it]
                                                       

 86%|████████▌ | 2570/2984 [7:21:58<1:10:11, 10.17s/it]
 86%|████████▌ | 2571/2984 [7:22:08<1:10:00, 10.17s/it]
 86%|████████▌ | 2572/2984 [7:22:18<1:10:25, 10.26s/it]
 86%|████████▌ | 2573/2984 [7:22:29<1:10:16, 10.26s/
0: {'loss': 0.4868, 'grad_norm': 0.897578483325493, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.52}
0: it]
 86%|████████▋ | 2574/2984 [7:22:39<1:10:01, 10.25s/it]
 86%|████████▋ | 2575/2984 [7:22:49<1:10:10, 10.29s/it]
 86%|████████▋ | 2576/2984 [7:23:00<1:10:07, 10.31s/it]
 86%|████████▋ | 2577/2984 [7:23:15<1:19:22, 11.70s/it]
 86%|████████▋ | 2578/2984 [7:23:28<1:21:28, 12.04s/it]
 86%|████████▋ | 2579/2984 [7:23:38<1:17:43, 11.51s/it]
 86%|████████▋ | 2580/2984 [7:23:48<1:15:21, 11.19s/it]
                                                       

 86%|████████▋ | 2580/2984 [7:23:49<1:15:21, 11.19s/it]
 86%|████████▋ | 2581/2984 [7:23:59<1:14:02, 11.02s/it]
 87%|████████▋ | 2582/2984 [7:24:09<1:12:24, 10.81s/it]
 87%|████████▋ | 2583/2984 [7:24:19<1:11:00, 10.62s/it]
 87%|████████▋ | 2584/2984 [7:24:30<1:10:01, 10.50s/it]
 87%|████████▋ | 2585/2984 [7:24:40<1:09:22, 10.43s/it]
 
0: {'loss': 0.5016, 'grad_norm': 0.9061484448865028, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.52}
0: 87%|████████▋ | 2586/2984 [7:24:50<1:08:47, 10.37s/it]
 87%|████████▋ | 2587/2984 [7:25:00<1:08:22, 10.33s/it]
 87%|████████▋ | 2588/2984 [7:25:11<1:07:59, 10.30s/it]
 87%|████████▋ | 2589/2984 [7:25:21<1:07:43, 10.29s/it]
 87%|████████▋ | 2590/2984 [7:25:31<1:07:30, 10.28s/it]
                                                       

 87%|████████▋ | 2590/2984 [7:25:31<1:07:30, 10.28s/it]
 87%|████████▋ | 2591/2984 [7:25:41<1:07:21, 10.28s/it]
 87%|████████▋ | 2592/2984 [7:25:52<1:07:07, 10.27s/it]
 87%|████████▋ | 2593/2984 [7:26:02<1:06:55, 10.27s/it]
 87%|████████▋ | 2594/2984 [7:26:12<1:06:38, 10.25s/it]
 87%|████████▋ | 2595/2984 [7:26:22<1:06:25, 10.25s/it]
 87%|████████▋ | 2596/2984 [7:26:33<1:06:14, 10.24s/it]
 87%|████████▋ | 2597/2984 [7:26:43<1:06:06, 10.25s/it]
 87%|�
0: {'loss': 0.4911, 'grad_norm': 0.8732341172782677, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.52}
0: ��███████▋ | 2598/2984 [7:26:53<1:05:52, 10.24s/it]
 87%|████████▋ | 2599/2984 [7:27:03<1:05:43, 10.24s/it]
 87%|████████▋ | 2600/2984 [7:27:14<1:05:39, 10.26s/it]
                                                       

 87%|████████▋ | 2600/2984 [7:27:14<1:05:39, 10.26s/it]
 87%|████████▋ | 2601/2984 [7:27:24<1:05:35, 10.27s/it]
 87%|████████▋ | 2602/2984 [7:27:34<1:05:22, 10.27s/it]
 87%|████████▋ | 2603/2984 [7:27:44<1:05:15, 10.28s/it]
 87%|████████▋ | 2604/2984 [7:27:55<1:05:04, 10.28s/it]
 87%|████████▋ | 2605/2984 [7:28:05<1:05:19, 10.34s/it]
 87%|████████▋ | 2606/2984 [7:28:16<1:05:39, 10.42s/it]
 87%|████████▋ | 2607/2984 [7:28:26<1:05:25, 10.41s/it]
 87%|████████▋ | 2608/2984 [7:28:37<1:05:13, 10.41s/it]
 87%|████████▋ | 2609/2984 [7:28:47<1:05:01, 10.40s/it]
 87%|██
0: {'loss': 0.493, 'grad_norm': 0.8321557815634646, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.52}
0: {'loss': 0.4958, 'grad_norm': 0.9143605120567058, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.53}
0: ██████▋ | 2610/2984 [7:28:57<1:04:49, 10.40s/it]
                                                       

 87%|████████▋ | 2610/2984 [7:28:57<1:04:49, 10.40s/it]
 88%|████████▊ | 2611/2984 [7:29:08<1:04:37, 10.40s/it]
 88%|████████▊ | 2612/2984 [7:29:18<1:04:25, 10.39s/it]
 88%|████████▊ | 2613/2984 [7:29:29<1:04:12, 10.39s/it]
 88%|████████▊ | 2614/2984 [7:29:39<1:04:01, 10.38s/it]
 88%|████████▊ | 2615/2984 [7:29:49<1:03:49, 10.38s/it]
 88%|████████▊ | 2616/2984 [7:30:00<1:04:02, 10.44s/it]
 88%|████████▊ | 2617/2984 [7:30:10<1:03:55, 10.45s/it]
 88%|████████▊ | 2618/2984 [7:30:21<1:03:45, 10.45s/it]
 88%|████████▊ | 2619/2984 [7:30:33<1:06:49, 10.99s/it]
 88%|████████▊ | 2620/2984 [7:30:43<1:05:38, 10.82s/it]
                                                       

 88%|████████▊ 
0: {'loss': 0.4987, 'grad_norm': 0.8372191763605417, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.53}
0: | 2620/2984 [7:30:43<1:05:38, 10.82s/it]
 88%|████████▊ | 2621/2984 [7:30:55<1:06:06, 10.93s/it]
 88%|████████▊ | 2622/2984 [7:31:05<1:05:06, 10.79s/it]
 88%|████████▊ | 2623/2984 [7:31:16<1:05:57, 10.96s/it]
 88%|████████▊ | 2624/2984 [7:31:27<1:04:35, 10.76s/it]
 88%|████████▊ | 2625/2984 [7:31:37<1:03:40, 10.64s/it]
 88%|████████▊ | 2626/2984 [7:31:47<1:02:42, 10.51s/it]
 88%|████████▊ | 2627/2984 [7:31:57<1:01:49, 10.39s/it]
 88%|████████▊ | 2628/2984 [7:32:08<1:01:51, 10.43s/it]
 88%|████████▊ | 2629/2984 [7:32:18<1:01:16, 10.36s/it]
 88%|████████▊ | 2630/2984 [7:32:28<1:00:40, 10.28s/it]
                                                       

 88%|████████▊ | 2630/2984 [7:32:28<1:00:40, 10.28s/it]
 88%|████████▊ | 2631/2984 [7:32:38<1:00:22, 10.26s/it]
 88%|████████▊ | 263
0: {'loss': 0.4872, 'grad_norm': 1.558762894585189, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.53}
0: 2/2984 [7:32:49<1:00:10, 10.26s/it]
 88%|████████▊ | 2633/2984 [7:32:59<1:00:09, 10.28s/it]
 88%|████████▊ | 2634/2984 [7:33:09<1:00:05, 10.30s/it]
 88%|████████▊ | 2635/2984 [7:33:20<59:58, 10.31s/it]  
 88%|████████▊ | 2636/2984 [7:33:30<59:44, 10.30s/it]
 88%|████████▊ | 2637/2984 [7:33:41<59:57, 10.37s/it]
 88%|████████▊ | 2638/2984 [7:33:51<59:25, 10.30s/it]
 88%|████████▊ | 2639/2984 [7:34:01<59:06, 10.28s/it]
 88%|████████▊ | 2640/2984 [7:34:11<59:17, 10.34s/it]
                                                     

 88%|████████▊ | 2640/2984 [7:34:11<59:17, 10.34s/it]
 89%|████████▊ | 2641/2984 [7:34:22<59:30, 10.41s/it]
 89%|████████▊ | 2642/2984 [7:34:33<59:38, 10.46s/it]
 89%|████████▊ | 2643/2984 [7:34:43<1:00:00, 10.56s/it]
 89%|████████▊ | 2644/2984 [7:34:54<59:47, 
0: {'loss': 0.4974, 'grad_norm': 0.7849551460945078, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.53}
0: 10.55s/it]  
 89%|████████▊ | 2645/2984 [7:35:04<59:33, 10.54s/it]
 89%|████████▊ | 2646/2984 [7:35:15<59:23, 10.54s/it]
 89%|████████▊ | 2647/2984 [7:35:25<58:55, 10.49s/it]
 89%|████████▊ | 2648/2984 [7:35:35<58:13, 10.40s/it]
 89%|████████▉ | 2649/2984 [7:35:46<57:36, 10.32s/it]
 89%|████████▉ | 2650/2984 [7:35:56<57:10, 10.27s/it]
                                                     

 89%|████████▉ | 2650/2984 [7:35:56<57:10, 10.27s/it]
 89%|████████▉ | 2651/2984 [7:36:06<56:48, 10.24s/it]
 89%|████████▉ | 2652/2984 [7:36:16<56:26, 10.20s/it]
 89%|████████▉ | 2653/2984 [7:36:26<56:03, 10.16s/it]
 89%|████████▉ | 2654/2984 [7:36:36<55:43, 10.13s/it]
 89%|████████▉ | 2655/2984 [7:36:46<55:28, 10.12s/it]
 89%|████████▉ | 2656/2984 [7:36:56<55:16, 10.11s/it]
 89%|█████
0: {'loss': 0.486, 'grad_norm': 0.8523461112640567, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.53}
0: ███▉ | 2657/2984 [7:37:07<55:17, 10.14s/it]
 89%|████████▉ | 2658/2984 [7:37:17<55:20, 10.19s/it]
 89%|████████▉ | 2659/2984 [7:37:27<55:13, 10.20s/it]
 89%|████████▉ | 2660/2984 [7:37:37<55:02, 10.19s/it]
                                                     

 89%|████████▉ | 2660/2984 [7:37:37<55:02, 10.19s/it]
 89%|████████▉ | 2661/2984 [7:37:47<54:52, 10.19s/it]
 89%|████████▉ | 2662/2984 [7:37:58<54:42, 10.19s/it]
 89%|████████▉ | 2663/2984 [7:38:08<54:47, 10.24s/it]
 89%|████████▉ | 2664/2984 [7:38:18<54:50, 10.28s/it]
 89%|████████▉ | 2665/2984 [7:38:29<54:47, 10.31s/it]
 89%|████████▉ | 2666/2984 [7:38:39<54:43, 10.33s/it]
 89%|████████▉ | 2667/2984 [7:38:49<54:21, 10.29s/it]
 89%|████████▉ | 2668/2984 [7:39:00<53:59, 10.25s/it]
 89%|████████▉ | 2669/2984 [7:39:10
0: {'loss': 0.4956, 'grad_norm': 0.7988894484857677, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.54}
0: {'loss': 0.4977, 'grad_norm': 1.3905925632669176, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.54}
0: <53:43, 10.23s/it]
 89%|████████▉ | 2670/2984 [7:39:20<53:22, 10.20s/it]
                                                     

 89%|████████▉ | 2670/2984 [7:39:20<53:22, 10.20s/it]
 90%|████████▉ | 2671/2984 [7:39:30<53:06, 10.18s/it]
 90%|████████▉ | 2672/2984 [7:39:40<53:01, 10.20s/it]
 90%|████████▉ | 2673/2984 [7:39:50<52:53, 10.20s/it]
 90%|████████▉ | 2674/2984 [7:40:01<52:44, 10.21s/it]
 90%|████████▉ | 2675/2984 [7:40:11<52:37, 10.22s/it]
 90%|████████▉ | 2676/2984 [7:40:21<52:31, 10.23s/it]
 90%|████████▉ | 2677/2984 [7:40:31<52:20, 10.23s/it]
 90%|████████▉ | 2678/2984 [7:40:42<52:17, 10.25s/it]
 90%|████████▉ | 2679/2984 [7:40:52<52:07, 10.25s/it]
 90%|████████▉ | 2680/2984 [7:41:02<51:53, 10.24s/it]
                                                     

 90%|████████�
0: {'loss': 0.4956, 'grad_norm': 0.8462968390664587, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.54}
0: � | 2680/2984 [7:41:02<51:53, 10.24s/it]
 90%|████████▉ | 2681/2984 [7:41:12<51:46, 10.25s/it]
 90%|████████▉ | 2682/2984 [7:41:23<51:35, 10.25s/it]
 90%|████████▉ | 2683/2984 [7:41:33<52:01, 10.37s/it]
 90%|████████▉ | 2684/2984 [7:41:44<52:57, 10.59s/it]
 90%|████████▉ | 2685/2984 [7:41:55<52:27, 10.53s/it]
 90%|█████████ | 2686/2984 [7:42:05<51:51, 10.44s/it]
 90%|█████████ | 2687/2984 [7:42:15<51:20, 10.37s/it]
 90%|█████████ | 2688/2984 [7:42:26<51:02, 10.34s/it]
 90%|█████████ | 2689/2984 [7:42:36<50:45, 10.32s/it]
 90%|█████████ | 2690/2984 [7:42:46<50:39, 10.34s/it]
                                                     

 90%|█████████ | 2690/2984 [7:42:46<50:39, 10.34s/it]
 90%|█████████ | 2691/2984 [7:42:56<50:10, 10.28s/it]
 90%|█████████ | 2692/2984 [7:43:07<49:56, 10.
0: {'loss': 0.4879, 'grad_norm': 1.3502934194968137, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.54}
0: 26s/it]
 90%|█████████ | 2693/2984 [7:43:17<49:50, 10.28s/it]
 90%|█████████ | 2694/2984 [7:43:27<49:48, 10.30s/it]
 90%|█████████ | 2695/2984 [7:43:38<49:47, 10.34s/it]
 90%|█████████ | 2696/2984 [7:43:48<49:39, 10.35s/it]
 90%|█████████ | 2697/2984 [7:43:58<49:27, 10.34s/it]
 90%|█████████ | 2698/2984 [7:44:09<49:14, 10.33s/it]
 90%|█████████ | 2699/2984 [7:44:19<49:02, 10.32s/it]
 90%|█████████ | 2700/2984 [7:44:29<48:49, 10.32s/it]
                                                     

 90%|█████████ | 2700/2984 [7:44:29<48:49, 10.32s/it]
 91%|█████████ | 2701/2984 [7:44:40<48:39, 10.32s/it]
 91%|█████████ | 2702/2984 [7:44:50<48:31, 10.32s/it]
 91%|█████████ | 2703/2984 [7:45:00<48:36, 10.38s/it]
 91%|█████████ | 2704/2984 [7:45:11<48:40, 10.43s/it]
 91%|██████�
0: {'loss': 0.487, 'grad_norm': 0.946340021328697, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.54}
0: �██ | 2705/2984 [7:45:21<48:37, 10.46s/it]
 91%|█████████ | 2706/2984 [7:45:32<48:26, 10.46s/it]
 91%|█████████ | 2707/2984 [7:45:42<48:15, 10.45s/it]
 91%|█████████ | 2708/2984 [7:45:53<47:59, 10.43s/it]
 91%|█████████ | 2709/2984 [7:46:03<47:33, 10.37s/it]
 91%|█████████ | 2710/2984 [7:46:13<47:13, 10.34s/it]
                                                     

 91%|█████████ | 2710/2984 [7:46:13<47:13, 10.34s/it]
 91%|█████████ | 2711/2984 [7:46:24<46:58, 10.32s/it]
 91%|█████████ | 2712/2984 [7:46:34<46:42, 10.30s/it]
 91%|█████████ | 2713/2984 [7:46:44<46:32, 10.30s/it]
 91%|█████████ | 2714/2984 [7:46:54<46:18, 10.29s/it]
 91%|█████████ | 2715/2984 [7:47:05<46:09, 10.30s/it]
 91%|█████████ | 2716/2984 [7:47:15<45:59, 10.30s/it]
 91%|█████████ | 2717/2984 [7:47:25<45:5
0: {'loss': 0.4873, 'grad_norm': 1.1368627015371169, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.55}
0: 4, 10.32s/it]
 91%|█████████ | 2718/2984 [7:47:36<45:46, 10.32s/it]
 91%|█████████ | 2719/2984 [7:47:46<45:33, 10.32s/it]
 91%|█████████ | 2720/2984 [7:47:56<45:24, 10.32s/it]
                                                     

 91%|█████████ | 2720/2984 [7:47:56<45:24, 10.32s/it]
 91%|█████████ | 2721/2984 [7:48:07<45:12, 10.32s/it]
 91%|█████████ | 2722/2984 [7:48:17<45:00, 10.31s/it]
 91%|█████████▏| 2723/2984 [7:48:27<44:53, 10.32s/it]
 91%|█████████▏| 2724/2984 [7:48:38<44:44, 10.32s/it]
 91%|█████████▏| 2725/2984 [7:48:48<44:37, 10.34s/it]
 91%|█████████▏| 2726/2984 [7:48:58<44:27, 10.34s/it]
 91%|█████████▏| 2727/2984 [7:49:09<44:18, 10.35s/it]
 91%|█████████▏| 2728/2984 [7:49:19<44:07, 10.34s/it]
 91%|█████████▏| 2729/2984 [7:49:29<43:56, 10.34s/it]
 91%|
0: {'loss': 0.4962, 'grad_norm': 0.9981984508430757, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.55}
0: {'loss': 0.4812, 'grad_norm': 0.9382061531601965, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.55}
0: █████████▏| 2730/2984 [7:49:40<43:47, 10.34s/it]
                                                     

 91%|█████████▏| 2730/2984 [7:49:40<43:47, 10.34s/it]
 92%|█████████▏| 2731/2984 [7:49:50<43:35, 10.34s/it]
 92%|█████████▏| 2732/2984 [7:50:00<43:27, 10.35s/it]
 92%|█████████▏| 2733/2984 [7:50:11<43:18, 10.35s/it]
 92%|█████████▏| 2734/2984 [7:50:21<43:13, 10.38s/it]
 92%|█████████▏| 2735/2984 [7:50:31<43:01, 10.37s/it]
 92%|█████████▏| 2736/2984 [7:50:42<42:49, 10.36s/it]
 92%|█████████▏| 2737/2984 [7:50:52<42:39, 10.36s/it]
 92%|█████████▏| 2738/2984 [7:51:03<42:30, 10.37s/it]
 92%|█████████▏| 2739/2984 [7:51:13<42:22, 10.38s/it]
 92%|█████████▏| 2740/2984 [7:51:23<42:06, 10.36s/it]
                                                     

 92%|████████�
0: {'loss': 0.4881, 'grad_norm': 0.8168145591845654, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.55}
0: �▏| 2740/2984 [7:51:23<42:06, 10.36s/it]
 92%|█████████▏| 2741/2984 [7:51:34<41:55, 10.35s/it]
 92%|█████████▏| 2742/2984 [7:51:44<41:49, 10.37s/it]
 92%|█████████▏| 2743/2984 [7:51:55<41:46, 10.40s/it]
 92%|█████████▏| 2744/2984 [7:52:05<41:26, 10.36s/it]
 92%|█████████▏| 2745/2984 [7:52:15<41:13, 10.35s/it]
 92%|█████████▏| 2746/2984 [7:52:25<41:00, 10.34s/it]
 92%|█████████▏| 2747/2984 [7:52:36<40:45, 10.32s/it]
 92%|█████████▏| 2748/2984 [7:52:46<40:36, 10.33s/it]
 92%|█████████▏| 2749/2984 [7:52:56<40:29, 10.34s/it]
 92%|█████████▏| 2750/2984 [7:53:07<40:19, 10.34s/it]
                                                     

 92%|█████████▏| 2750/2984 [7:53:07<40:19, 10.34s/it]
 92%|█████████▏| 2751/2984 [7:53:17<40:10, 10.35s/it]
 92%|█████████▏| 2
0: {'loss': 0.4904, 'grad_norm': 1.1618893965066934, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.55}
0: 752/2984 [7:53:27<39:57, 10.34s/it]
 92%|█████████▏| 2753/2984 [7:53:38<39:43, 10.32s/it]
 92%|█████████▏| 2754/2984 [7:53:48<39:27, 10.29s/it]
 92%|█████████▏| 2755/2984 [7:53:58<39:06, 10.25s/it]
 92%|█████████▏| 2756/2984 [7:54:08<38:47, 10.21s/it]
 92%|█████████▏| 2757/2984 [7:54:19<38:53, 10.28s/it]
 92%|█████████▏| 2758/2984 [7:54:29<38:36, 10.25s/it]
 92%|█████████▏| 2759/2984 [7:54:39<38:20, 10.22s/it]
 92%|█████████▏| 2760/2984 [7:54:49<38:10, 10.23s/it]
                                                     

 92%|█████████▏| 2760/2984 [7:54:49<38:10, 10.23s/it]
 93%|█████████▎| 2761/2984 [7:54:59<37:59, 10.22s/it]
 93%|█████████▎| 2762/2984 [7:55:10<37:52, 10.23s/it]
 93%|█████████▎| 2763/2984 [7:55:20<37:43, 10.24s/it]
 93%|█████████▎| 2764/298
0: {'loss': 0.4941, 'grad_norm': 0.8197506530570241, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.56}
0: 4 [7:55:30<37:33, 10.25s/it]
 93%|█████████▎| 2765/2984 [7:55:40<37:24, 10.25s/it]
 93%|█████████▎| 2766/2984 [7:55:51<37:16, 10.26s/it]
 93%|█████████▎| 2767/2984 [7:56:01<37:10, 10.28s/it]
 93%|█████████▎| 2768/2984 [7:56:11<36:54, 10.25s/it]
 93%|█████████▎| 2769/2984 [7:56:21<36:35, 10.21s/it]
 93%|█████████▎| 2770/2984 [7:56:32<36:22, 10.20s/it]
                                                     

 93%|█████████▎| 2770/2984 [7:56:32<36:22, 10.20s/it]
 93%|█████████▎| 2771/2984 [7:56:42<36:14, 10.21s/it]
 93%|█████████▎| 2772/2984 [7:56:52<36:04, 10.21s/it]
 93%|█████████▎| 2773/2984 [7:57:02<35:57, 10.23s/it]
 93%|█████████▎| 2774/2984 [7:57:13<35:49, 10.24s/it]
 93%|█████████▎| 2775/2984 [7:57:23<35:42, 10.25s/it]
 93%|█████████▎| 2776/2984 [7:57
0: {'loss': 0.4814, 'grad_norm': 0.8719053841074187, 'learning_rate': 5e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.56}
0: :33<35:34, 10.26s/it]
 93%|█████████▎| 2777/2984 [7:57:43<35:22, 10.25s/it]
 93%|█████████▎| 2778/2984 [7:57:54<35:12, 10.26s/it]
 93%|█████████▎| 2779/2984 [7:58:04<34:59, 10.24s/it]
 93%|█████████▎| 2780/2984 [7:58:14<34:47, 10.23s/it]
                                                     

 93%|█████████▎| 2780/2984 [7:58:14<34:47, 10.23s/it]
 93%|█████████▎| 2781/2984 [7:58:24<34:37, 10.24s/it]
 93%|█████████▎| 2782/2984 [7:58:35<34:42, 10.31s/it]
 93%|█████████▎| 2783/2984 [7:58:45<34:30, 10.30s/it]
 93%|█████████▎| 2784/2984 [7:58:55<34:16, 10.28s/it]
 93%|█████████▎| 2785/2984 [7:59:05<34:02, 10.26s/it]
 93%|█████████▎| 2786/2984 [7:59:16<33:55, 10.28s/it]
 93%|█████████▎| 2787/2984 [7:59:26<33:49, 10.30s/it]
 93%|█████████▎| 2788/2984 [7:59:37<33:
0: {'loss': 0.4914, 'grad_norm': 0.8135836520210976, 'learning_rate': 4.993064000899538e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.56}
0: 47, 10.35s/it]
 93%|█████████▎| 2789/2984 [7:59:47<33:37, 10.35s/it]
 93%|█████████▎| 2790/2984 [7:59:57<33:25, 10.34s/it]
                                                     

 93%|█████████▎| 2790/2984 [7:59:57<33:25, 10.34s/it]
 94%|█████████▎| 2791/2984 [8:00:08<33:15, 10.34s/it]
 94%|█████████▎| 2792/2984 [8:00:18<33:05, 10.34s/it]
 94%|█████████▎| 2793/2984 [8:00:28<32:52, 10.33s/it]
 94%|█████████▎| 2794/2984 [8:00:39<32:45, 10.35s/it]
 94%|█████████▎| 2795/2984 [8:00:49<32:38, 10.36s/it]
 94%|█████████▎| 2796/2984 [8:00:59<32:21, 10.33s/it]
 94%|█████████▎| 2797/2984 [8:01:10<32:10, 10.32s/it]
 94%|█████████▍| 2798/2984 [8:01:20<31:51, 10.28s/it]
 94%|█████████▍| 2799/2984 [8:01:30<31:35, 10.24s/it]
 94%|█████████▍| 2800/2984 [8:01:40<31:20, 10.
0: {'loss': 0.4872, 'grad_norm': 0.9374224055417277, 'learning_rate': 4.9378323208947725e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.56}
0: {'loss': 0.4872, 'grad_norm': 0.8688510804326383, 'learning_rate': 4.828728948150395e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.56}
0: 22s/it]
                                                     

 94%|█████████▍| 2800/2984 [8:01:40<31:20, 10.22s/it]
 94%|█████████▍| 2801/2984 [8:01:50<31:09, 10.22s/it]
 94%|█████████▍| 2802/2984 [8:02:00<30:56, 10.20s/it]
 94%|█████████▍| 2803/2984 [8:02:11<30:42, 10.18s/it]
 94%|█████████▍| 2804/2984 [8:02:21<30:37, 10.21s/it]
 94%|█████████▍| 2805/2984 [8:02:31<30:26, 10.20s/it]
 94%|█████████▍| 2806/2984 [8:02:41<30:16, 10.21s/it]
 94%|█████████▍| 2807/2984 [8:02:51<30:05, 10.20s/it]
 94%|█████████▍| 2808/2984 [8:03:02<29:54, 10.19s/it]
 94%|█████████▍| 2809/2984 [8:03:12<29:45, 10.20s/it]
 94%|█████████▍| 2810/2984 [8:03:22<29:37, 10.21s/it]
                                                     

 94%|█████████▍| 2810/2984 [8:03:22<29:37, 10.21s/it]
 94%|████�
0: {'loss': 0.4897, 'grad_norm': 0.9275607371353127, 'learning_rate': 4.668440369796708e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.57}
0: ��████▍| 2811/2984 [8:03:32<29:29, 10.23s/it]
 94%|█████████▍| 2812/2984 [8:03:43<29:22, 10.25s/it]
 94%|█████████▍| 2813/2984 [8:03:53<29:09, 10.23s/it]
 94%|█████████▍| 2814/2984 [8:04:03<29:01, 10.24s/it]
 94%|█████████▍| 2815/2984 [8:04:13<28:49, 10.23s/it]
 94%|█████████▍| 2816/2984 [8:04:24<28:41, 10.25s/it]
 94%|█████████▍| 2817/2984 [8:04:34<28:32, 10.25s/it]
 94%|█████████▍| 2818/2984 [8:04:44<28:19, 10.24s/it]
 94%|█████████▍| 2819/2984 [8:04:54<28:07, 10.23s/it]
 95%|█████████▍| 2820/2984 [8:05:04<27:57, 10.23s/it]
                                                     

 95%|█████████▍| 2820/2984 [8:05:05<27:57, 10.23s/it]
 95%|█████████▍| 2821/2984 [8:05:15<27:50, 10.25s/it]
 95%|█████████▍| 2822/2984 [8:05:25<27:38, 10.24s/it]
 95%|██████�
0: {'loss': 0.5024, 'grad_norm': 0.886210021408103, 'learning_rate': 4.46091342260007e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.57}
0: �██▍| 2823/2984 [8:05:35<27:25, 10.22s/it]
 95%|█████████▍| 2824/2984 [8:05:45<27:14, 10.21s/it]
 95%|█████████▍| 2825/2984 [8:05:56<27:04, 10.22s/it]
 95%|█████████▍| 2826/2984 [8:06:06<26:52, 10.21s/it]
 95%|█████████▍| 2827/2984 [8:06:16<26:39, 10.19s/it]
 95%|█████████▍| 2828/2984 [8:06:26<26:27, 10.17s/it]
 95%|█████████▍| 2829/2984 [8:06:36<26:15, 10.16s/it]
 95%|█████████▍| 2830/2984 [8:06:46<26:03, 10.15s/it]
                                                     

 95%|█████████▍| 2830/2984 [8:06:46<26:03, 10.15s/it]
 95%|█████████▍| 2831/2984 [8:06:57<25:53, 10.16s/it]
 95%|█████████▍| 2832/2984 [8:07:07<25:40, 10.13s/it]
 95%|█████████▍| 2833/2984 [8:07:17<25:25, 10.10s/it]
 95%|█████████▍| 2834/2984 [8:07:27<25:17, 10.12s/it]
 95%|█████████
0: {'loss': 0.4904, 'grad_norm': 0.7918050439976424, 'learning_rate': 4.211258108742913e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.57}
0: ▌| 2835/2984 [8:07:37<25:16, 10.18s/it]
 95%|█████████▌| 2836/2984 [8:07:47<25:14, 10.24s/it]
 95%|█████████▌| 2837/2984 [8:07:58<25:06, 10.25s/it]
 95%|█████████▌| 2838/2984 [8:08:08<25:01, 10.29s/it]
 95%|█████████▌| 2839/2984 [8:08:19<24:57, 10.33s/it]
 95%|█████████▌| 2840/2984 [8:08:29<24:52, 10.36s/it]
                                                     

 95%|█████████▌| 2840/2984 [8:08:29<24:52, 10.36s/it]
 95%|█████████▌| 2841/2984 [8:08:39<24:46, 10.39s/it]
 95%|█████████▌| 2842/2984 [8:08:50<24:36, 10.40s/it]
 95%|█████████▌| 2843/2984 [8:09:00<24:29, 10.42s/it]
 95%|█████████▌| 2844/2984 [8:09:11<24:19, 10.43s/it]
 95%|█████████▌| 2845/2984 [8:09:21<24:06, 10.41s/it]
 95%|█████████▌| 2846/2984 [8:09:31<23:52, 10.38s/it]
 95%|█████████▌| 28
0: {'loss': 0.4822, 'grad_norm': 0.8208220745250224, 'learning_rate': 3.925621770610885e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.57}
0: 47/2984 [8:09:42<23:34, 10.33s/it]
 95%|█████████▌| 2848/2984 [8:09:52<23:20, 10.29s/it]
 95%|█████████▌| 2849/2984 [8:10:02<23:08, 10.28s/it]
 96%|█████████▌| 2850/2984 [8:10:12<22:57, 10.28s/it]
                                                     

 96%|█████████▌| 2850/2984 [8:10:12<22:57, 10.28s/it]
 96%|█████████▌| 2851/2984 [8:10:23<22:48, 10.29s/it]
 96%|█████████▌| 2852/2984 [8:10:33<22:36, 10.28s/it]
 96%|█████████▌| 2853/2984 [8:10:43<22:25, 10.27s/it]
 96%|█████████▌| 2854/2984 [8:10:53<22:12, 10.25s/it]
 96%|█████████▌| 2855/2984 [8:11:04<21:59, 10.23s/it]
 96%|█████████▌| 2856/2984 [8:11:14<21:46, 10.20s/it]
 96%|█████████▌| 2857/2984 [8:11:24<21:31, 10.17s/it]
 96%|█████████▌| 2858/2984 [8:11:34<21:16, 10.13s/it]
 96%|█████████▌| 2859/2984
0: {'loss': 0.483, 'grad_norm': 1.0705681524214343, 'learning_rate': 3.611037722821452e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.57}
0:  [8:11:44<21:04, 10.11s/it]
 96%|█████████▌| 2860/2984 [8:11:54<20:52, 10.10s/it]
                                                     

 96%|█████████▌| 2860/2984 [8:11:54<20:52, 10.10s/it]
 96%|█████████▌| 2861/2984 [8:12:04<20:42, 10.10s/it]
 96%|█████████▌| 2862/2984 [8:12:14<20:31, 10.10s/it]
 96%|█████████▌| 2863/2984 [8:12:24<20:20, 10.09s/it]
 96%|█████████▌| 2864/2984 [8:12:34<20:09, 10.08s/it]
 96%|█████████▌| 2865/2984 [8:12:44<19:57, 10.06s/it]
 96%|█████████▌| 2866/2984 [8:12:54<19:45, 10.04s/it]
 96%|█████████▌| 2867/2984 [8:13:04<19:34, 10.03s/it]
 96%|█████████▌| 2868/2984 [8:13:14<19:22, 10.02s/it]
 96%|█████████▌| 2869/2984 [8:13:24<19:11, 10.01s/it]
 96%|█████████▌| 2870/2984 [8:13:35<19:06, 10.05s/it]
                                                     
0: {'loss': 0.4871, 'grad_norm': 0.8405530238830633, 'learning_rate': 3.275252068675788e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.58}
0: {'loss': 0.4853, 'grad_norm': 0.8895702392394971, 'learning_rate': 2.9265329653876517e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.58}
0: 

 96%|█████████▌| 2870/2984 [8:13:35<19:06, 10.05s/it]
 96%|█████████▌| 2871/2984 [8:13:45<18:55, 10.05s/it]
 96%|█████████▌| 2872/2984 [8:13:55<18:42, 10.03s/it]
 96%|█████████▋| 2873/2984 [8:14:05<18:36, 10.06s/it]
 96%|█████████▋| 2874/2984 [8:14:15<18:32, 10.12s/it]
 96%|█████████▋| 2875/2984 [8:14:25<18:28, 10.17s/it]
 96%|█████████▋| 2876/2984 [8:14:36<18:22, 10.21s/it]
 96%|█████████▋| 2877/2984 [8:14:46<18:15, 10.24s/it]
 96%|█████████▋| 2878/2984 [8:14:56<18:06, 10.25s/it]
 96%|█████████▋| 2879/2984 [8:15:06<17:51, 10.21s/it]
 97%|█████████▋| 2880/2984 [8:15:16<17:38, 10.18s/it]
                                                     

 97%|█████████▋| 2880/2984 [8:15:16<17:38, 10.18s/it]
 97%|█████████▋| 2881/2984 [8:15:26<17:26, 10.16s/it]
 97%|
0: {'loss': 0.4895, 'grad_norm': 0.936636389392958, 'learning_rate': 2.57346703461235e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.58}
0: █████████▋| 2882/2984 [8:15:37<17:14, 10.14s/it]
 97%|█████████▋| 2883/2984 [8:15:47<17:01, 10.12s/it]
 97%|█████████▋| 2884/2984 [8:15:57<16:49, 10.10s/it]
 97%|█████████▋| 2885/2984 [8:16:07<16:40, 10.10s/it]
 97%|█████████▋| 2886/2984 [8:16:17<16:31, 10.11s/it]
 97%|█████████▋| 2887/2984 [8:16:27<16:23, 10.14s/it]
 97%|█████████▋| 2888/2984 [8:16:37<16:14, 10.15s/it]
 97%|█████████▋| 2889/2984 [8:16:47<16:06, 10.17s/it]
 97%|█████████▋| 2890/2984 [8:16:58<15:56, 10.17s/it]
                                                     

 97%|█████████▋| 2890/2984 [8:16:58<15:56, 10.17s/it]
 97%|█████████▋| 2891/2984 [8:17:08<15:46, 10.18s/it]
 97%|█████████▋| 2892/2984 [8:17:18<15:37, 10.19s/it]
 97%|█████████▋| 2893/2984 [8:17:28<15:26, 10.19s/it]
 97%|██�
0: {'loss': 0.481, 'grad_norm': 0.7905723397877862, 'learning_rate': 2.2247479313242137e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.58}
0: ��██████▋| 2894/2984 [8:17:39<15:22, 10.25s/it]
 97%|█████████▋| 2895/2984 [8:17:49<15:12, 10.26s/it]
 97%|█████████▋| 2896/2984 [8:17:59<14:57, 10.20s/it]
 97%|█████████▋| 2897/2984 [8:18:09<14:40, 10.13s/it]
 97%|█████████▋| 2898/2984 [8:18:19<14:27, 10.08s/it]
 97%|█████████▋| 2899/2984 [8:18:29<14:14, 10.05s/it]
 97%|█████████▋| 2900/2984 [8:18:39<14:03, 10.04s/it]
                                                     

 97%|█████████▋| 2900/2984 [8:18:39<14:03, 10.04s/it]
 97%|█████████▋| 2901/2984 [8:18:49<13:52, 10.03s/it]
 97%|█████████▋| 2902/2984 [8:18:59<13:44, 10.06s/it]
 97%|█████████▋| 2903/2984 [8:19:09<13:36, 10.08s/it]
 97%|█████████▋| 2904/2984 [8:19:19<13:28, 10.11s/it]
 97%|█████████▋| 2905/2984 [8:19:29<13:19, 10.12s/it]
 97%|████�
0: {'loss': 0.4832, 'grad_norm': 0.8609622275685551, 'learning_rate': 1.888962277178548e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.59}
0: �████▋| 2906/2984 [8:19:40<13:09, 10.12s/it]
 97%|█████████▋| 2907/2984 [8:19:50<13:00, 10.14s/it]
 97%|█████████▋| 2908/2984 [8:20:00<12:49, 10.13s/it]
 97%|█████████▋| 2909/2984 [8:20:10<12:40, 10.14s/it]
 98%|█████████▊| 2910/2984 [8:20:20<12:31, 10.15s/it]
                                                     

 98%|█████████▊| 2910/2984 [8:20:20<12:31, 10.15s/it]
 98%|█████████▊| 2911/2984 [8:20:30<12:20, 10.14s/it]
 98%|█████████▊| 2912/2984 [8:20:40<12:09, 10.13s/it]
 98%|█████████▊| 2913/2984 [8:20:51<11:58, 10.12s/it]
 98%|█████████▊| 2914/2984 [8:21:01<11:50, 10.15s/it]
 98%|█████████▊| 2915/2984 [8:21:11<11:42, 10.18s/it]
 98%|█████████▊| 2916/2984 [8:21:21<11:36, 10.24s/it]
 98%|█████████▊| 2917/2984 [8:21:32<11:31, 10.32s/it]
 98%|███████
0: {'loss': 0.4867, 'grad_norm': 0.813165306048767, 'learning_rate': 1.5743782293891153e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.59}
0: ██▊| 2918/2984 [8:21:42<11:24, 10.38s/it]
 98%|█████████▊| 2919/2984 [8:21:53<11:17, 10.42s/it]
 98%|█████████▊| 2920/2984 [8:22:03<11:08, 10.45s/it]
                                                     

 98%|█████████▊| 2920/2984 [8:22:03<11:08, 10.45s/it]
 98%|█████████▊| 2921/2984 [8:22:14<10:59, 10.47s/it]
 98%|█████████▊| 2922/2984 [8:22:25<10:50, 10.49s/it]
 98%|█████████▊| 2923/2984 [8:22:35<10:40, 10.50s/it]
 98%|█████████▊| 2924/2984 [8:22:46<10:29, 10.49s/it]
 98%|█████████▊| 2925/2984 [8:22:56<10:18, 10.49s/it]
 98%|█████████▊| 2926/2984 [8:23:06<10:04, 10.43s/it]
 98%|█████████▊| 2927/2984 [8:23:17<09:51, 10.37s/it]
 98%|█████████▊| 2928/2984 [8:23:27<09:36, 10.30s/it]
 98%|█████████▊| 2929/2984 [8:23:37<09:25, 10.28s/it]
 98%|█████████�
0: {'loss': 0.4895, 'grad_norm': 0.7732004620296122, 'learning_rate': 1.288741891257087e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.59}
0: {'loss': 0.4802, 'grad_norm': 0.7670023217683808, 'learning_rate': 1.0390865773999307e-06, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.59}
0: ��| 2930/2984 [8:23:47<09:15, 10.29s/it]
                                                     

 98%|█████████▊| 2930/2984 [8:23:47<09:15, 10.29s/it]
 98%|█████████▊| 2931/2984 [8:23:57<09:00, 10.19s/it]
 98%|█████████▊| 2932/2984 [8:24:07<08:49, 10.18s/it]
 98%|█████████▊| 2933/2984 [8:24:18<08:41, 10.22s/it]
 98%|█████████▊| 2934/2984 [8:24:28<08:31, 10.22s/it]
 98%|█████████▊| 2935/2984 [8:24:38<08:20, 10.22s/it]
 98%|█████████▊| 2936/2984 [8:24:48<08:10, 10.22s/it]
 98%|█████████▊| 2937/2984 [8:24:58<07:59, 10.20s/it]
 98%|█████████▊| 2938/2984 [8:25:09<07:49, 10.20s/it]
 98%|█████████▊| 2939/2984 [8:25:19<07:38, 10.19s/it]
 99%|█████████▊| 2940/2984 [8:25:29<07:28, 10.19s/it]
                                                     

 99%|█████████▊| 2940/2984 [8:25:29<07:
0: {'loss': 0.4864, 'grad_norm': 0.8084439679296679, 'learning_rate': 8.315596302032928e-07, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.59}
0: 28, 10.19s/it]
 99%|█████████▊| 2941/2984 [8:25:39<07:18, 10.20s/it]
 99%|█████████▊| 2942/2984 [8:25:49<07:06, 10.15s/it]
 99%|█████████▊| 2943/2984 [8:25:59<06:54, 10.12s/it]
 99%|█████████▊| 2944/2984 [8:26:09<06:43, 10.08s/it]
 99%|█████████▊| 2945/2984 [8:26:19<06:31, 10.05s/it]
 99%|█████████▊| 2946/2984 [8:26:29<06:21, 10.05s/it]
 99%|█████████▉| 2947/2984 [8:26:39<06:11, 10.05s/it]
 99%|█████████▉| 2948/2984 [8:26:49<06:01, 10.05s/it]
 99%|█████████▉| 2949/2984 [8:27:00<05:52, 10.06s/it]
 99%|█████████▉| 2950/2984 [8:27:10<05:42, 10.07s/it]
                                                     

 99%|█████████▉| 2950/2984 [8:27:10<05:42, 10.07s/it]
 99%|█████████▉| 2951/2984 [8:27:20<05:31, 10.06s/it]
 99%|█████████▉| 2952/2984 [8:27:30<05:21, 10.
0: {'loss': 0.4801, 'grad_norm': 1.0183954211000086, 'learning_rate': 6.712710518496049e-07, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.6}
0: 05s/it]
 99%|█████████▉| 2953/2984 [8:27:40<05:11, 10.05s/it]
 99%|█████████▉| 2954/2984 [8:27:50<05:06, 10.20s/it]
 99%|█████████▉| 2955/2984 [8:28:01<04:59, 10.31s/it]
 99%|█████████▉| 2956/2984 [8:28:11<04:49, 10.33s/it]
 99%|█████████▉| 2957/2984 [8:28:22<04:40, 10.41s/it]
 99%|█████████▉| 2958/2984 [8:28:32<04:30, 10.41s/it]
 99%|█████████▉| 2959/2984 [8:28:42<04:19, 10.38s/it]
 99%|█████████▉| 2960/2984 [8:28:53<04:08, 10.34s/it]
                                                     

 99%|█████████▉| 2960/2984 [8:28:53<04:08, 10.34s/it]
 99%|█████████▉| 2961/2984 [8:29:03<03:59, 10.41s/it]
 99%|█████████▉| 2962/2984 [8:29:14<03:48, 10.41s/it]
 99%|█████████▉| 2963/2984 [8:29:24<03:38, 10.39s/it]
 99%|█████████▉| 2964/2984 [8:29:34<03:27, 10.37s/it]
0: {'loss': 0.4996, 'grad_norm': 0.7346294921782509, 'learning_rate': 5.621676791052278e-07, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.6}
0: 
 99%|█████████▉| 2965/2984 [8:29:45<03:16, 10.36s/it]
 99%|█████████▉| 2966/2984 [8:29:55<03:06, 10.34s/it]
 99%|█████████▉| 2967/2984 [8:30:05<02:55, 10.34s/it]
 99%|█████████▉| 2968/2984 [8:30:16<02:45, 10.33s/it]
 99%|█████████▉| 2969/2984 [8:30:26<02:34, 10.32s/it]
100%|█████████▉| 2970/2984 [8:30:36<02:24, 10.34s/it]
                                                     

100%|█████████▉| 2970/2984 [8:30:36<02:24, 10.34s/it]
100%|█████████▉| 2971/2984 [8:30:47<02:14, 10.35s/it]
100%|█████████▉| 2972/2984 [8:30:57<02:04, 10.35s/it]
100%|█████████▉| 2973/2984 [8:31:07<01:53, 10.34s/it]
100%|█████████▉| 2974/2984 [8:31:18<01:43, 10.33s/it]
100%|█████████▉| 2975/2984 [8:31:28<01:32, 10.32s/it]
100%|█████████▉| 2976/2984 [8:31:38<01:22, 10.31s/it]
100%|�
0: {'loss': 0.4907, 'grad_norm': 0.7443092102031795, 'learning_rate': 5.069359991004621e-07, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.6}
0: [2025-11-23 23:26:05,810] [INFO] [axolotl.core.trainers.base._save:613] [PID:2356327] [RANK:0] Saving model checkpoint to /lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-4b/0.75/checkpoint-2984
0: [2025-11-23 23:26:13,108] [INFO] [axolotl.core.trainers.base._save:662] [PID:2356327] [RANK:0] Saving Trainer.data_collator.tokenizer by default as Trainer.processing_class is `None`
0: {'train_runtime': 30801.9908, 'train_samples_per_second': 1.55, 'train_steps_per_second': 0.097, 'train_loss': 0.52924768382998, 'memory/max_mem_active(gib)': 57.09, 'memory/max_mem_allocated(gib)': 57.09, 'memory/device_mem_reserved(gib)': 66.93, 'epoch': 0.6}
0: ��████████▉| 2977/2984 [8:31:49<01:12, 10.30s/it]
100%|█████████▉| 2978/2984 [8:31:59<01:01, 10.30s/it]
100%|█████████▉| 2979/2984 [8:32:09<00:51, 10.28s/it]
100%|█████████▉| 2980/2984 [8:32:19<00:41, 10.27s/it]
                                                     

100%|█████████▉| 2980/2984 [8:32:19<00:41, 10.27s/it]
100%|█████████▉| 2981/2984 [8:32:30<00:30, 10.29s/it]
100%|█████████▉| 2982/2984 [8:32:40<00:20, 10.30s/it]
100%|█████████▉| 2983/2984 [8:32:50<00:10, 10.32s/it]
100%|██████████| 2984/2984 [8:33:01<00:00, 10.31s/it]
                                                     

100%|██████████| 2984/2984 [8:33:21<00:00, 10.31s/it]
100%|██████████| 2984/2984 [8:33:21<00:00, 10.32s/it]
0: [2025-11-23 23:26:24,347] [INFO] [axolotl.train.save_trained_model:228] [PID:2356327] [RANK:0] Training completed! Saving trained model to /lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-4b/0.75.
0: [2025-11-23 23:26:36,215] [INFO] [axolotl.core.trainers.base._save:613] [PID:2356327] [RANK:0] Saving model checkpoint to /lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-4b/0.75
0: jzxh014:2356327:2358134 [0] NCCL INFO [Service thread] Connection closed by localRank 3
0: jzxh014:2356327:2357247 [0] NCCL INFO [Service thread] Connection closed by localRank 3
0: jzxh014:2356327:2358134 [0] NCCL INFO [Service thread] Connection closed by localRank 1
0: jzxh014:2356327:2357247 [0] NCCL INFO [Service thread] Connection closed by localRank 1
0: jzxh014:2356327:2358134 [0] NCCL INFO [Service thread] Connection closed by localRank 2
0: jzxh014:2356327:2357247 [0] NCCL INFO [Service thread] Connection closed by localRank 2
0: [2025-11-23 23:26:43,193] [INFO] [axolotl.core.trainers.base._save:662] [PID:2356327] [RANK:0] Saving Trainer.data_collator.tokenizer by default as Trainer.processing_class is `None`
0: [2025-11-23 23:26:43,804] [INFO] [axolotl.train.save_trained_model:350] [PID:2356327] [RANK:0] Model successfully saved to /lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-4b/0.75