================================================== 开始评估:任务=triviaqa | 少样本数=0 | 模型=Qwen2.5-32B 输出路径:results2/Qwen2.5-32B/base_triviaqa.json ================================================== The following values were not passed to `accelerate launch` and had defaults used instead: More than one GPU was found, enabling multi-GPU training. If this was unintended please pass in `--num_processes=1`. `--num_machines` was set to a value of `1` `--mixed_precision` was set to a value of `'no'` `--dynamo_backend` was set to a value of `'no'` To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`. 2025-12-01:21:15:37 INFO [__main__:440] Selected Tasks: ['triviaqa'] 2025-12-01:21:15:37 INFO [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2025-12-01:21:15:37 INFO [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-32B'} 2025-12-01:21:15:37 INFO [__main__:440] Selected Tasks: ['triviaqa'] 2025-12-01:21:15:37 INFO [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2025-12-01:21:15:37 INFO [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-32B'} 2025-12-01:21:15:38 WARNING [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. 2025-12-01:21:15:38 INFO [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'} `torch_dtype` is deprecated! Use `dtype` instead! 2025-12-01:21:15:38 INFO [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'} Loading checkpoint shards: 0%| | 0/17 [00:00 n136-128-154:806117:806706 [0] NCCL INFO Initialized NET plugin IB n136-128-154:806117:806706 [0] NCCL INFO Assigned NET plugin IB to comm n136-128-154:806117:806706 [0] NCCL INFO Using network IB n136-128-154:806118:806707 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. n136-128-154:806118:806707 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0> n136-128-154:806118:806707 [1] NCCL INFO Initialized NET plugin IB n136-128-154:806118:806707 [1] NCCL INFO Assigned NET plugin IB to comm n136-128-154:806118:806707 [1] NCCL INFO Using network IB n136-128-154:806118:806707 [1] NCCL INFO ncclCommInitRankConfig comm 0x1019b9f0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xf12573ebaaf36444 - Init START n136-128-154:806117:806706 [0] NCCL INFO ncclCommInitRankConfig comm 0x101fba50 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xf12573ebaaf36444 - Init START n136-128-154:806118:806707 [1] NCCL INFO RAS client listening socket at ::1<28028> n136-128-154:806117:806706 [0] NCCL INFO RAS client listening socket at ::1<28028> n136-128-154:806118:806707 [1] NCCL INFO TOPO/NET : Importing network plugins to topology n136-128-154:806118:806707 [1] NCCL INFO Retrieving state for IB n136-128-154:806118:806707 [1] NCCL INFO Initialized state 0 for IB n136-128-154:806118:806707 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null) n136-128-154:806117:806706 [0] NCCL INFO TOPO/NET : Importing network plugins to topology n136-128-154:806117:806706 [0] NCCL INFO Retrieving state for IB n136-128-154:806117:806706 [0] NCCL INFO Initialized state 0 for IB n136-128-154:806117:806706 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null) n136-128-154:806118:806707 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null) n136-128-154:806117:806706 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null) n136-128-154:806118:806707 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null) n136-128-154:806117:806706 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null) n136-128-154:806118:806707 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null) n136-128-154:806117:806706 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null) n136-128-154:806118:806707 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default n136-128-154:806118:806707 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default n136-128-154:806117:806706 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default n136-128-154:806117:806706 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default n136-128-154:806118:806707 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 === n136-128-154:806118:806707 [1] NCCL INFO CPU/0-1 (1/1/2) n136-128-154:806118:806707 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff) n136-128-154:806118:806707 [1] NCCL INFO + PCI[24.0] - NIC/0-95000 n136-128-154:806118:806707 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff) n136-128-154:806118:806707 [1] NCCL INFO + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8) n136-128-154:806118:806707 [1] NCCL INFO + PCI[24.0] - GPU/0-c5000 (0) n136-128-154:806118:806707 [1] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:806118:806707 [1] NCCL INFO + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8) n136-128-154:806118:806707 [1] NCCL INFO + PCI[24.0] - GPU/0-c9000 (1) n136-128-154:806118:806707 [1] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:806118:806707 [1] NCCL INFO + PCI[24.0] - NIC/0-cc000 n136-128-154:806118:806707 [1] NCCL INFO + SYS[10.0] - CPU/0-0 n136-128-154:806118:806707 [1] NCCL INFO CPU/0-0 (1/1/2) n136-128-154:806118:806707 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff) n136-128-154:806118:806707 [1] NCCL INFO + PCI[24.0] - NIC/0-1d000 n136-128-154:806118:806707 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff) n136-128-154:806118:806707 [1] NCCL INFO + PCI[24.0] - NIC/0-60000 n136-128-154:806118:806707 [1] NCCL INFO + SYS[10.0] - CPU/0-1 n136-128-154:806118:806707 [1] NCCL INFO ========================================== n136-128-154:806118:806707 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:806118:806707 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:806118:806707 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126 n136-128-154:806117:806706 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 === n136-128-154:806117:806706 [0] NCCL INFO CPU/0-1 (1/1/2) n136-128-154:806117:806706 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff) n136-128-154:806117:806706 [0] NCCL INFO + PCI[24.0] - NIC/0-95000 n136-128-154:806117:806706 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff) n136-128-154:806117:806706 [0] NCCL INFO + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8) n136-128-154:806117:806706 [0] NCCL INFO + PCI[24.0] - GPU/0-c5000 (0) n136-128-154:806117:806706 [0] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:806117:806706 [0] NCCL INFO + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8) n136-128-154:806117:806706 [0] NCCL INFO + PCI[24.0] - GPU/0-c9000 (1) n136-128-154:806117:806706 [0] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:806117:806706 [0] NCCL INFO + PCI[24.0] - NIC/0-cc000 n136-128-154:806117:806706 [0] NCCL INFO + SYS[10.0] - CPU/0-0 n136-128-154:806117:806706 [0] NCCL INFO CPU/0-0 (1/1/2) n136-128-154:806117:806706 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff) n136-128-154:806117:806706 [0] NCCL INFO + PCI[24.0] - NIC/0-1d000 n136-128-154:806117:806706 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff) n136-128-154:806117:806706 [0] NCCL INFO + PCI[24.0] - NIC/0-60000 n136-128-154:806117:806706 [0] NCCL INFO + SYS[10.0] - CPU/0-1 n136-128-154:806117:806706 [0] NCCL INFO ========================================== n136-128-154:806117:806706 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:806117:806706 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:806118:806707 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1 n136-128-154:806118:806707 [1] NCCL INFO 0 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806117:806706 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126 n136-128-154:806118:806707 [1] NCCL INFO 1 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806118:806707 [1] NCCL INFO 2 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806118:806707 [1] NCCL INFO 3 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806118:806707 [1] NCCL INFO 4 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806118:806707 [1] NCCL INFO 5 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806118:806707 [1] NCCL INFO 6 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806118:806707 [1] NCCL INFO 7 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806118:806707 [1] NCCL INFO 8 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806118:806707 [1] NCCL INFO 9 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806118:806707 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806118:806707 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806118:806707 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0 n136-128-154:806118:806707 [1] NCCL INFO 0 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806118:806707 [1] NCCL INFO 1 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806118:806707 [1] NCCL INFO 2 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806118:806707 [1] NCCL INFO 3 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806118:806707 [1] NCCL INFO 4 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806118:806707 [1] NCCL INFO 5 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806118:806707 [1] NCCL INFO 6 : GPU/0-c9000 GPU/0-c5000 n136-128-154:806118:806707 [1] NCCL INFO 7 : GPU/0-c9000 GPU/0-c5000 n136-128-154:806118:806707 [1] NCCL INFO 8 : GPU/0-c9000 GPU/0-c5000 n136-128-154:806118:806707 [1] NCCL INFO 9 : GPU/0-c9000 GPU/0-c5000 n136-128-154:806118:806707 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000 n136-128-154:806118:806707 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000 n136-128-154:806117:806706 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1 n136-128-154:806117:806706 [0] NCCL INFO 0 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806117:806706 [0] NCCL INFO 1 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806117:806706 [0] NCCL INFO 2 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806117:806706 [0] NCCL INFO 3 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806117:806706 [0] NCCL INFO 4 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806117:806706 [0] NCCL INFO 5 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806117:806706 [0] NCCL INFO 6 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806117:806706 [0] NCCL INFO 7 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806117:806706 [0] NCCL INFO 8 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806117:806706 [0] NCCL INFO 9 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806117:806706 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806117:806706 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806117:806706 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0 n136-128-154:806117:806706 [0] NCCL INFO 0 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806117:806706 [0] NCCL INFO 1 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806117:806706 [0] NCCL INFO 2 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806117:806706 [0] NCCL INFO 3 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806117:806706 [0] NCCL INFO 4 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806117:806706 [0] NCCL INFO 5 : GPU/0-c5000 GPU/0-c9000 n136-128-154:806117:806706 [0] NCCL INFO 6 : GPU/0-c9000 GPU/0-c5000 n136-128-154:806117:806706 [0] NCCL INFO 7 : GPU/0-c9000 GPU/0-c5000 n136-128-154:806117:806706 [0] NCCL INFO 8 : GPU/0-c9000 GPU/0-c5000 n136-128-154:806117:806706 [0] NCCL INFO 9 : GPU/0-c9000 GPU/0-c5000 n136-128-154:806117:806706 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000 n136-128-154:806117:806706 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000 n136-128-154:806118:806707 [1] NCCL INFO comm 0x1019b9f0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0 n136-128-154:806117:806706 [0] NCCL INFO comm 0x101fba50 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0 n136-128-154:806118:806707 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1 n136-128-154:806118:806707 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1 n136-128-154:806118:806707 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1 n136-128-154:806118:806707 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1 n136-128-154:806118:806707 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1 n136-128-154:806118:806707 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1 n136-128-154:806118:806707 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1 n136-128-154:806118:806707 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1 n136-128-154:806118:806707 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1 n136-128-154:806118:806707 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1 n136-128-154:806118:806707 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1 n136-128-154:806118:806707 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1 n136-128-154:806118:806707 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1 n136-128-154:806118:806707 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1 n136-128-154:806118:806707 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1 n136-128-154:806118:806707 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1 n136-128-154:806118:806707 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1 n136-128-154:806118:806707 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1 n136-128-154:806118:806707 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1 n136-128-154:806118:806707 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1 n136-128-154:806118:806707 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1 n136-128-154:806118:806707 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1 n136-128-154:806118:806707 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1 n136-128-154:806118:806707 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1 n136-128-154:806117:806706 [0] NCCL INFO Channel 00/24 : 0 1 n136-128-154:806118:806707 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0 n136-128-154:806117:806706 [0] NCCL INFO Channel 01/24 : 0 1 n136-128-154:806118:806707 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0 n136-128-154:806117:806706 [0] NCCL INFO Channel 02/24 : 0 1 n136-128-154:806118:806707 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0 n136-128-154:806117:806706 [0] NCCL INFO Channel 03/24 : 0 1 n136-128-154:806118:806707 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0 n136-128-154:806117:806706 [0] NCCL INFO Channel 04/24 : 0 1 n136-128-154:806118:806707 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0 n136-128-154:806117:806706 [0] NCCL INFO Channel 05/24 : 0 1 n136-128-154:806118:806707 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0 n136-128-154:806117:806706 [0] NCCL INFO Channel 06/24 : 0 1 n136-128-154:806118:806707 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0 n136-128-154:806117:806706 [0] NCCL INFO Channel 07/24 : 0 1 n136-128-154:806118:806707 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0 n136-128-154:806117:806706 [0] NCCL INFO Channel 08/24 : 0 1 n136-128-154:806118:806707 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0 n136-128-154:806117:806706 [0] NCCL INFO Channel 09/24 : 0 1 n136-128-154:806118:806707 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0 n136-128-154:806117:806706 [0] NCCL INFO Channel 10/24 : 0 1 n136-128-154:806118:806707 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0 n136-128-154:806117:806706 [0] NCCL INFO Channel 11/24 : 0 1 n136-128-154:806118:806707 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0 n136-128-154:806117:806706 [0] NCCL INFO Channel 12/24 : 0 1 n136-128-154:806118:806707 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0 n136-128-154:806117:806706 [0] NCCL INFO Channel 13/24 : 0 1 n136-128-154:806118:806707 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0 n136-128-154:806117:806706 [0] NCCL INFO Channel 14/24 : 0 1 n136-128-154:806118:806707 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0 n136-128-154:806117:806706 [0] NCCL INFO Channel 15/24 : 0 1 n136-128-154:806118:806707 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0 n136-128-154:806117:806706 [0] NCCL INFO Channel 16/24 : 0 1 n136-128-154:806118:806707 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0 n136-128-154:806117:806706 [0] NCCL INFO Channel 17/24 : 0 1 n136-128-154:806118:806707 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0 n136-128-154:806117:806706 [0] NCCL INFO Channel 18/24 : 0 1 n136-128-154:806118:806707 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0 n136-128-154:806117:806706 [0] NCCL INFO Channel 19/24 : 0 1 n136-128-154:806118:806707 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0 n136-128-154:806117:806706 [0] NCCL INFO Channel 20/24 : 0 1 n136-128-154:806118:806707 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0 n136-128-154:806117:806706 [0] NCCL INFO Channel 21/24 : 0 1 n136-128-154:806118:806707 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0 n136-128-154:806117:806706 [0] NCCL INFO Channel 22/24 : 0 1 n136-128-154:806118:806707 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0 n136-128-154:806117:806706 [0] NCCL INFO Channel 23/24 : 0 1 n136-128-154:806118:806707 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0 n136-128-154:806118:806707 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1 n136-128-154:806117:806706 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1 n136-128-154:806117:806706 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1 n136-128-154:806118:806707 [1] NCCL INFO P2P Chunksize set to 524288 n136-128-154:806117:806706 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1 n136-128-154:806117:806706 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1 n136-128-154:806117:806706 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1 n136-128-154:806117:806706 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1 n136-128-154:806117:806706 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1 n136-128-154:806117:806706 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1 n136-128-154:806117:806706 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1 n136-128-154:806117:806706 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1 n136-128-154:806117:806706 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1 n136-128-154:806117:806706 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1 n136-128-154:806117:806706 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1 n136-128-154:806117:806706 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1 n136-128-154:806117:806706 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1 n136-128-154:806117:806706 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1 n136-128-154:806117:806706 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1 n136-128-154:806117:806706 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1 n136-128-154:806117:806706 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1 n136-128-154:806117:806706 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1 n136-128-154:806117:806706 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1 n136-128-154:806117:806706 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1 n136-128-154:806117:806706 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1 n136-128-154:806117:806706 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1 n136-128-154:806117:806706 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1 n136-128-154:806117:806706 [0] NCCL INFO P2P Chunksize set to 524288 n136-128-154:806117:806706 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. n136-128-154:806117:806706 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 n136-128-154:806117:806729 [0] NCCL INFO [Proxy Service] Device 0 CPU core 99 n136-128-154:806117:806730 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 100 n136-128-154:806118:806707 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. n136-128-154:806118:806731 [1] NCCL INFO [Proxy Service] Device 1 CPU core 110 n136-128-154:806118:806732 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 47 n136-128-154:806117:806706 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 n136-128-154:806117:806706 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer n136-128-154:806117:806706 [0] NCCL INFO CC Off, workFifoBytes 1048576 n136-128-154:806118:806707 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 n136-128-154:806118:806707 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer n136-128-154:806117:806706 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. n136-128-154:806117:806706 [0] NCCL INFO ncclCommInitRankConfig comm 0x101fba50 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xf12573ebaaf36444 - Init COMPLETE n136-128-154:806117:806706 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.77 (kernels 0.14, alloc 0.40, bootstrap 0.00, allgathers 0.01, topo 0.08, graphs 0.00, connections 0.03, rest 0.11) n136-128-154:806118:806707 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. n136-128-154:806118:806707 [1] NCCL INFO ncclCommInitRankConfig comm 0x1019b9f0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xf12573ebaaf36444 - Init COMPLETE n136-128-154:806118:806707 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.77 (kernels 0.14, alloc 0.40, bootstrap 0.00, allgathers 0.01, topo 0.08, graphs 0.00, connections 0.03, rest 0.11) n136-128-154:806117:806734 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:806117:806734 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:806117:806734 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806117:806734 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806117:806734 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806117:806734 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806117:806734 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806117:806734 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806117:806734 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806117:806734 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806117:806734 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806117:806734 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806117:806734 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806117:806734 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806117:806734 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806117:806734 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806117:806734 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806117:806734 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806117:806734 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806117:806734 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806117:806734 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806117:806734 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806117:806734 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806117:806734 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:806735 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 n136-128-154:806117:806734 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 2025-12-01:21:16:40 INFO [evaluator:559] Running generate_until requests 2025-12-01:21:16:40 INFO [evaluator:559] Running generate_until requests Running generate_until requests: 0%| | 0/8972 [00:00 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:806118:815804 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization' To add an exception for this directory, call: git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization n136-128-154:806118:815834 [1] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:806118:815834 [1] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:806118:815834 [1] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:806118:815834 [1] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:806118:815834 [1] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:806118:815834 [1] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:806118:806731 [1] NCCL INFO misc/socket.cc:915 -> 3 2025-12-01:21:53:28 INFO [loggers.evaluation_tracker:209] Saving results aggregated hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-32B), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto | Tasks |Version| Filter |n-shot| Metric | |Value| |Stderr| |--------|------:|-----------------|-----:|-----------|---|----:|---|-----:| |triviaqa| 3|remove_whitespace| 0|exact_match|↑ |0.618|± |0.0036| [rank0]:[W1201 21:53:31.956965897 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) n136-128-154:806117:815882 [0] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:806117:815882 [0] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:806117:815882 [0] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:806117:806729 [0] NCCL INFO misc/socket.cc:915 -> 3 n136-128-154:806117:815882 [0] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:806117:815882 [0] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:806117:815882 [0] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:806118:806731 [1] NCCL INFO misc/socket.cc:915 -> 3 n136-128-154:806118:815834 [1] NCCL INFO comm 0x1019b9f0 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE n136-128-154:806117:815882 [0] NCCL INFO comm 0x101fba50 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE 任务 triviaqa 评估完成! ================================================== 开始评估:任务=winogrande | 少样本数=5 | 模型=Qwen2.5-32B 输出路径:results2/Qwen2.5-32B/base_winogrande.json ================================================== The following values were not passed to `accelerate launch` and had defaults used instead: More than one GPU was found, enabling multi-GPU training. If this was unintended please pass in `--num_processes=1`. `--num_machines` was set to a value of `1` `--mixed_precision` was set to a value of `'no'` `--dynamo_backend` was set to a value of `'no'` To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`. 2025-12-01:21:54:22 INFO [__main__:440] Selected Tasks: ['winogrande'] 2025-12-01:21:54:22 INFO [__main__:440] Selected Tasks: ['winogrande'] 2025-12-01:21:54:22 INFO [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2025-12-01:21:54:22 INFO [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-32B'} 2025-12-01:21:54:22 INFO [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2025-12-01:21:54:22 INFO [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-32B'} 2025-12-01:21:54:23 WARNING [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. 2025-12-01:21:54:23 INFO [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'} 2025-12-01:21:54:23 INFO [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'} `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! Loading checkpoint shards: 0%| | 0/17 [00:00 n136-128-154:815929:816165 [0] NCCL INFO Initialized NET plugin IB n136-128-154:815929:816165 [0] NCCL INFO Assigned NET plugin IB to comm n136-128-154:815929:816165 [0] NCCL INFO Using network IB n136-128-154:815929:816165 [0] NCCL INFO ncclCommInitRankConfig comm 0x11eeb740 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xd3225c942fe0c8ed - Init START 2025-12-01:21:55:09 WARNING [evaluator:309] Overwriting default num_fewshot of winogrande from None to 5 2025-12-01:21:55:09 INFO [api.task:434] Building contexts for winogrande on rank 1... 0%| | 0/633 [00:00 n136-128-154:815930:816187 [1] NCCL INFO Initialized NET plugin IB n136-128-154:815930:816187 [1] NCCL INFO Assigned NET plugin IB to comm n136-128-154:815930:816187 [1] NCCL INFO Using network IB n136-128-154:815930:816187 [1] NCCL INFO ncclCommInitRankConfig comm 0x11f46e80 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xd3225c942fe0c8ed - Init START n136-128-154:815930:816187 [1] NCCL INFO RAS client listening socket at ::1<28028> n136-128-154:815929:816165 [0] NCCL INFO RAS client listening socket at ::1<28028> n136-128-154:815930:816187 [1] NCCL INFO TOPO/NET : Importing network plugins to topology n136-128-154:815930:816187 [1] NCCL INFO Retrieving state for IB n136-128-154:815930:816187 [1] NCCL INFO Initialized state 0 for IB n136-128-154:815930:816187 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null) n136-128-154:815929:816165 [0] NCCL INFO TOPO/NET : Importing network plugins to topology n136-128-154:815929:816165 [0] NCCL INFO Retrieving state for IB n136-128-154:815929:816165 [0] NCCL INFO Initialized state 0 for IB n136-128-154:815930:816187 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null) n136-128-154:815929:816165 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null) n136-128-154:815930:816187 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null) n136-128-154:815929:816165 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null) n136-128-154:815930:816187 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null) n136-128-154:815929:816165 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null) n136-128-154:815929:816165 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null) n136-128-154:815930:816187 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default n136-128-154:815930:816187 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default n136-128-154:815929:816165 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default n136-128-154:815929:816165 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default n136-128-154:815929:816165 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 === n136-128-154:815929:816165 [0] NCCL INFO CPU/0-1 (1/1/2) n136-128-154:815929:816165 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff) n136-128-154:815929:816165 [0] NCCL INFO + PCI[24.0] - NIC/0-95000 n136-128-154:815929:816165 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff) n136-128-154:815929:816165 [0] NCCL INFO + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8) n136-128-154:815929:816165 [0] NCCL INFO + PCI[24.0] - GPU/0-c5000 (0) n136-128-154:815929:816165 [0] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:815929:816165 [0] NCCL INFO + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8) n136-128-154:815929:816165 [0] NCCL INFO + PCI[24.0] - GPU/0-c9000 (1) n136-128-154:815929:816165 [0] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:815929:816165 [0] NCCL INFO + PCI[24.0] - NIC/0-cc000 n136-128-154:815929:816165 [0] NCCL INFO + SYS[10.0] - CPU/0-0 n136-128-154:815929:816165 [0] NCCL INFO CPU/0-0 (1/1/2) n136-128-154:815929:816165 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff) n136-128-154:815929:816165 [0] NCCL INFO + PCI[24.0] - NIC/0-1d000 n136-128-154:815929:816165 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff) n136-128-154:815929:816165 [0] NCCL INFO + PCI[24.0] - NIC/0-60000 n136-128-154:815929:816165 [0] NCCL INFO + SYS[10.0] - CPU/0-1 n136-128-154:815929:816165 [0] NCCL INFO ========================================== n136-128-154:815929:816165 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:815929:816165 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:815929:816165 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126 n136-128-154:815930:816187 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 === n136-128-154:815930:816187 [1] NCCL INFO CPU/0-1 (1/1/2) n136-128-154:815930:816187 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff) n136-128-154:815930:816187 [1] NCCL INFO + PCI[24.0] - NIC/0-95000 n136-128-154:815930:816187 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff) n136-128-154:815930:816187 [1] NCCL INFO + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8) n136-128-154:815930:816187 [1] NCCL INFO + PCI[24.0] - GPU/0-c5000 (0) n136-128-154:815930:816187 [1] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:815930:816187 [1] NCCL INFO + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8) n136-128-154:815930:816187 [1] NCCL INFO + PCI[24.0] - GPU/0-c9000 (1) n136-128-154:815930:816187 [1] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:815930:816187 [1] NCCL INFO + PCI[24.0] - NIC/0-cc000 n136-128-154:815930:816187 [1] NCCL INFO + SYS[10.0] - CPU/0-0 n136-128-154:815930:816187 [1] NCCL INFO CPU/0-0 (1/1/2) n136-128-154:815930:816187 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff) n136-128-154:815930:816187 [1] NCCL INFO + PCI[24.0] - NIC/0-1d000 n136-128-154:815930:816187 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff) n136-128-154:815930:816187 [1] NCCL INFO + PCI[24.0] - NIC/0-60000 n136-128-154:815930:816187 [1] NCCL INFO + SYS[10.0] - CPU/0-1 n136-128-154:815930:816187 [1] NCCL INFO ========================================== n136-128-154:815930:816187 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:815930:816187 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:815930:816187 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126 n136-128-154:815929:816165 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1 n136-128-154:815929:816165 [0] NCCL INFO 0 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815929:816165 [0] NCCL INFO 1 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815929:816165 [0] NCCL INFO 2 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815929:816165 [0] NCCL INFO 3 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815929:816165 [0] NCCL INFO 4 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815929:816165 [0] NCCL INFO 5 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815929:816165 [0] NCCL INFO 6 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815929:816165 [0] NCCL INFO 7 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815929:816165 [0] NCCL INFO 8 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815929:816165 [0] NCCL INFO 9 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815929:816165 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815929:816165 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815929:816165 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0 n136-128-154:815929:816165 [0] NCCL INFO 0 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815929:816165 [0] NCCL INFO 1 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815929:816165 [0] NCCL INFO 2 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815929:816165 [0] NCCL INFO 3 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815929:816165 [0] NCCL INFO 4 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815929:816165 [0] NCCL INFO 5 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815929:816165 [0] NCCL INFO 6 : GPU/0-c9000 GPU/0-c5000 n136-128-154:815929:816165 [0] NCCL INFO 7 : GPU/0-c9000 GPU/0-c5000 n136-128-154:815929:816165 [0] NCCL INFO 8 : GPU/0-c9000 GPU/0-c5000 n136-128-154:815929:816165 [0] NCCL INFO 9 : GPU/0-c9000 GPU/0-c5000 n136-128-154:815929:816165 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000 n136-128-154:815929:816165 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000 n136-128-154:815930:816187 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1 n136-128-154:815930:816187 [1] NCCL INFO 0 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815930:816187 [1] NCCL INFO 1 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815930:816187 [1] NCCL INFO 2 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815930:816187 [1] NCCL INFO 3 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815930:816187 [1] NCCL INFO 4 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815930:816187 [1] NCCL INFO 5 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815930:816187 [1] NCCL INFO 6 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815930:816187 [1] NCCL INFO 7 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815930:816187 [1] NCCL INFO 8 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815930:816187 [1] NCCL INFO 9 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815930:816187 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815930:816187 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815930:816187 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0 n136-128-154:815930:816187 [1] NCCL INFO 0 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815930:816187 [1] NCCL INFO 1 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815930:816187 [1] NCCL INFO 2 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815930:816187 [1] NCCL INFO 3 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815930:816187 [1] NCCL INFO 4 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815930:816187 [1] NCCL INFO 5 : GPU/0-c5000 GPU/0-c9000 n136-128-154:815930:816187 [1] NCCL INFO 6 : GPU/0-c9000 GPU/0-c5000 n136-128-154:815930:816187 [1] NCCL INFO 7 : GPU/0-c9000 GPU/0-c5000 n136-128-154:815930:816187 [1] NCCL INFO 8 : GPU/0-c9000 GPU/0-c5000 n136-128-154:815930:816187 [1] NCCL INFO 9 : GPU/0-c9000 GPU/0-c5000 n136-128-154:815930:816187 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000 n136-128-154:815930:816187 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000 n136-128-154:815929:816165 [0] NCCL INFO comm 0x11eeb740 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0 n136-128-154:815930:816187 [1] NCCL INFO comm 0x11f46e80 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0 n136-128-154:815929:816165 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1 n136-128-154:815930:816187 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1 n136-128-154:815929:816165 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1 n136-128-154:815930:816187 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1 n136-128-154:815929:816165 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1 n136-128-154:815930:816187 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1 n136-128-154:815929:816165 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1 n136-128-154:815930:816187 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1 n136-128-154:815929:816165 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1 n136-128-154:815930:816187 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1 n136-128-154:815929:816165 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1 n136-128-154:815930:816187 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1 n136-128-154:815929:816165 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1 n136-128-154:815930:816187 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1 n136-128-154:815929:816165 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1 n136-128-154:815930:816187 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1 n136-128-154:815929:816165 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1 n136-128-154:815930:816187 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1 n136-128-154:815929:816165 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1 n136-128-154:815930:816187 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1 n136-128-154:815929:816165 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1 n136-128-154:815930:816187 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1 n136-128-154:815929:816165 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1 n136-128-154:815930:816187 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1 n136-128-154:815929:816165 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1 n136-128-154:815930:816187 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1 n136-128-154:815929:816165 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1 n136-128-154:815930:816187 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1 n136-128-154:815929:816165 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1 n136-128-154:815930:816187 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1 n136-128-154:815929:816165 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1 n136-128-154:815930:816187 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1 n136-128-154:815929:816165 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1 n136-128-154:815930:816187 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1 n136-128-154:815929:816165 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1 n136-128-154:815930:816187 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1 n136-128-154:815929:816165 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1 n136-128-154:815930:816187 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1 n136-128-154:815929:816165 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1 n136-128-154:815930:816187 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1 n136-128-154:815929:816165 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1 n136-128-154:815930:816187 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1 n136-128-154:815929:816165 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1 n136-128-154:815930:816187 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1 n136-128-154:815929:816165 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1 n136-128-154:815930:816187 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1 n136-128-154:815929:816165 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1 n136-128-154:815930:816187 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1 n136-128-154:815929:816165 [0] NCCL INFO Channel 00/24 : 0 1 n136-128-154:815929:816165 [0] NCCL INFO Channel 01/24 : 0 1 n136-128-154:815929:816165 [0] NCCL INFO Channel 02/24 : 0 1 n136-128-154:815929:816165 [0] NCCL INFO Channel 03/24 : 0 1 n136-128-154:815929:816165 [0] NCCL INFO Channel 04/24 : 0 1 n136-128-154:815930:816187 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0 n136-128-154:815929:816165 [0] NCCL INFO Channel 05/24 : 0 1 n136-128-154:815930:816187 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0 n136-128-154:815929:816165 [0] NCCL INFO Channel 06/24 : 0 1 n136-128-154:815930:816187 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0 n136-128-154:815929:816165 [0] NCCL INFO Channel 07/24 : 0 1 n136-128-154:815930:816187 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0 n136-128-154:815929:816165 [0] NCCL INFO Channel 08/24 : 0 1 n136-128-154:815930:816187 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0 n136-128-154:815929:816165 [0] NCCL INFO Channel 09/24 : 0 1 n136-128-154:815930:816187 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0 n136-128-154:815929:816165 [0] NCCL INFO Channel 10/24 : 0 1 n136-128-154:815930:816187 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0 n136-128-154:815929:816165 [0] NCCL INFO Channel 11/24 : 0 1 n136-128-154:815930:816187 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0 n136-128-154:815929:816165 [0] NCCL INFO Channel 12/24 : 0 1 n136-128-154:815930:816187 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0 n136-128-154:815929:816165 [0] NCCL INFO Channel 13/24 : 0 1 n136-128-154:815930:816187 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0 n136-128-154:815929:816165 [0] NCCL INFO Channel 14/24 : 0 1 n136-128-154:815930:816187 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0 n136-128-154:815929:816165 [0] NCCL INFO Channel 15/24 : 0 1 n136-128-154:815930:816187 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0 n136-128-154:815929:816165 [0] NCCL INFO Channel 16/24 : 0 1 n136-128-154:815930:816187 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0 n136-128-154:815929:816165 [0] NCCL INFO Channel 17/24 : 0 1 n136-128-154:815930:816187 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0 n136-128-154:815929:816165 [0] NCCL INFO Channel 18/24 : 0 1 n136-128-154:815930:816187 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0 n136-128-154:815929:816165 [0] NCCL INFO Channel 19/24 : 0 1 n136-128-154:815930:816187 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0 n136-128-154:815929:816165 [0] NCCL INFO Channel 20/24 : 0 1 n136-128-154:815930:816187 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0 n136-128-154:815929:816165 [0] NCCL INFO Channel 21/24 : 0 1 n136-128-154:815929:816165 [0] NCCL INFO Channel 22/24 : 0 1 n136-128-154:815930:816187 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0 n136-128-154:815930:816187 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0 n136-128-154:815929:816165 [0] NCCL INFO Channel 23/24 : 0 1 n136-128-154:815930:816187 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0 n136-128-154:815929:816165 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1 n136-128-154:815930:816187 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0 n136-128-154:815929:816165 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1 n136-128-154:815930:816187 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0 n136-128-154:815929:816165 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1 n136-128-154:815930:816187 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0 n136-128-154:815929:816165 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1 n136-128-154:815930:816187 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0 n136-128-154:815929:816165 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1 n136-128-154:815930:816187 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1 n136-128-154:815929:816165 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1 n136-128-154:815929:816165 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1 n136-128-154:815930:816187 [1] NCCL INFO P2P Chunksize set to 524288 n136-128-154:815929:816165 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1 n136-128-154:815929:816165 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1 n136-128-154:815929:816165 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1 n136-128-154:815929:816165 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1 n136-128-154:815929:816165 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1 n136-128-154:815929:816165 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1 n136-128-154:815929:816165 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1 n136-128-154:815929:816165 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1 n136-128-154:815929:816165 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1 n136-128-154:815929:816165 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1 n136-128-154:815929:816165 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1 n136-128-154:815929:816165 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1 n136-128-154:815929:816165 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1 n136-128-154:815929:816165 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1 n136-128-154:815929:816165 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1 n136-128-154:815929:816165 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1 n136-128-154:815929:816165 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1 n136-128-154:815929:816165 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1 n136-128-154:815929:816165 [0] NCCL INFO P2P Chunksize set to 524288 n136-128-154:815930:816187 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. n136-128-154:815930:816195 [1] NCCL INFO [Proxy Service] Device 1 CPU core 99 n136-128-154:815930:816196 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 36 n136-128-154:815929:816165 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. n136-128-154:815929:816165 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 n136-128-154:815929:816198 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 115 n136-128-154:815929:816197 [0] NCCL INFO [Proxy Service] Device 0 CPU core 114 n136-128-154:815930:816187 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 n136-128-154:815930:816187 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer n136-128-154:815929:816165 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 n136-128-154:815929:816165 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer n136-128-154:815929:816165 [0] NCCL INFO CC Off, workFifoBytes 1048576 n136-128-154:815929:816165 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. n136-128-154:815929:816165 [0] NCCL INFO ncclCommInitRankConfig comm 0x11eeb740 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xd3225c942fe0c8ed - Init COMPLETE n136-128-154:815929:816165 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 8.65 (kernels 0.13, alloc 0.13, bootstrap 8.15, allgathers 0.01, topo 0.09, graphs 0.00, connections 0.02, rest 0.13) n136-128-154:815930:816187 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. n136-128-154:815930:816187 [1] NCCL INFO ncclCommInitRankConfig comm 0x11f46e80 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xd3225c942fe0c8ed - Init COMPLETE n136-128-154:815930:816187 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.65 (kernels 0.12, alloc 0.29, bootstrap 0.00, allgathers 0.01, topo 0.09, graphs 0.00, connections 0.02, rest 0.13) n136-128-154:815929:816199 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:815929:816199 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:815929:816199 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:815929:816199 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:815929:816199 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:815929:816199 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:815929:816199 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:815929:816199 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:815929:816199 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815929:816199 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815929:816199 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815929:816199 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815929:816199 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815929:816199 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815929:816199 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815929:816199 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815929:816199 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815929:816199 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815929:816199 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815929:816199 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815929:816199 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815929:816199 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815929:816199 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815929:816199 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816200 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 n136-128-154:815929:816199 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 2025-12-01:21:55:10 INFO [evaluator:559] Running loglikelihood requests 2025-12-01:21:55:10 INFO [evaluator:559] Running loglikelihood requests Running loglikelihood requests: 0%| | 0/1268 [00:00 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:815930:816278 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization' To add an exception for this directory, call: git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization n136-128-154:815930:816285 [1] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:815930:816285 [1] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:815930:816285 [1] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:815930:816285 [1] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:815930:816285 [1] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:815930:816285 [1] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:815930:816195 [1] NCCL INFO misc/socket.cc:915 -> 3 2025-12-01:21:56:33 INFO [loggers.evaluation_tracker:209] Saving results aggregated hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-32B), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: auto (64) | Tasks |Version|Filter|n-shot|Metric| |Value | |Stderr| |----------|------:|------|-----:|------|---|-----:|---|-----:| |winogrande| 1|none | 5|acc |↑ |0.8106|± | 0.011| [rank0]:[W1201 21:56:34.591631452 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) n136-128-154:815929:816335 [0] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:815929:816335 [0] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:815929:816335 [0] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:815929:816197 [0] NCCL INFO misc/socket.cc:915 -> 3 n136-128-154:815929:816335 [0] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:815929:816335 [0] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:815929:816335 [0] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:815930:816195 [1] NCCL INFO misc/socket.cc:915 -> 3 n136-128-154:815930:816285 [1] NCCL INFO comm 0x11f46e80 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE n136-128-154:815929:816335 [0] NCCL INFO comm 0x11eeb740 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE 任务 winogrande 评估完成! ================================================== 开始评估:任务=mmlu | 少样本数=5 | 模型=Qwen2.5-32B 输出路径:results2/Qwen2.5-32B/base_mmlu.json ================================================== The following values were not passed to `accelerate launch` and had defaults used instead: More than one GPU was found, enabling multi-GPU training. If this was unintended please pass in `--num_processes=1`. `--num_machines` was set to a value of `1` `--mixed_precision` was set to a value of `'no'` `--dynamo_backend` was set to a value of `'no'` To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`. 2025-12-01:21:57:24 INFO [__main__:440] Selected Tasks: ['mmlu'] 2025-12-01:21:57:24 INFO [__main__:440] Selected Tasks: ['mmlu'] 2025-12-01:21:57:24 INFO [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2025-12-01:21:57:24 INFO [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-32B'} 2025-12-01:21:57:24 INFO [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2025-12-01:21:57:24 INFO [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-32B'} 2025-12-01:21:57:25 WARNING [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. 2025-12-01:21:57:25 INFO [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'} `torch_dtype` is deprecated! Use `dtype` instead! 2025-12-01:21:57:25 INFO [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'} `torch_dtype` is deprecated! Use `dtype` instead! Loading checkpoint shards: 0%| | 0/17 [00:00 n136-128-154:816381:818575 [0] NCCL INFO Initialized NET plugin IB n136-128-154:816382:818576 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. n136-128-154:816382:818576 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0> n136-128-154:816382:818576 [1] NCCL INFO Initialized NET plugin IB n136-128-154:816381:818575 [0] NCCL INFO Assigned NET plugin IB to comm n136-128-154:816381:818575 [0] NCCL INFO Using network IB n136-128-154:816382:818576 [1] NCCL INFO Assigned NET plugin IB to comm n136-128-154:816382:818576 [1] NCCL INFO Using network IB n136-128-154:816381:818575 [0] NCCL INFO ncclCommInitRankConfig comm 0x11c77940 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x14739c438e31f4a1 - Init START n136-128-154:816382:818576 [1] NCCL INFO ncclCommInitRankConfig comm 0x11a96e00 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x14739c438e31f4a1 - Init START n136-128-154:816381:818575 [0] NCCL INFO RAS client listening socket at ::1<28028> n136-128-154:816382:818576 [1] NCCL INFO RAS client listening socket at ::1<28028> n136-128-154:816382:818576 [1] NCCL INFO TOPO/NET : Importing network plugins to topology n136-128-154:816382:818576 [1] NCCL INFO Retrieving state for IB n136-128-154:816382:818576 [1] NCCL INFO Initialized state 0 for IB n136-128-154:816382:818576 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null) n136-128-154:816381:818575 [0] NCCL INFO TOPO/NET : Importing network plugins to topology n136-128-154:816381:818575 [0] NCCL INFO Retrieving state for IB n136-128-154:816381:818575 [0] NCCL INFO Initialized state 0 for IB n136-128-154:816382:818576 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null) n136-128-154:816381:818575 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null) n136-128-154:816381:818575 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null) n136-128-154:816382:818576 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null) n136-128-154:816382:818576 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null) n136-128-154:816381:818575 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null) n136-128-154:816381:818575 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null) n136-128-154:816381:818575 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default n136-128-154:816381:818575 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default n136-128-154:816382:818576 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default n136-128-154:816382:818576 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default n136-128-154:816381:818575 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 === n136-128-154:816381:818575 [0] NCCL INFO CPU/0-1 (1/1/2) n136-128-154:816381:818575 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff) n136-128-154:816381:818575 [0] NCCL INFO + PCI[24.0] - NIC/0-95000 n136-128-154:816381:818575 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff) n136-128-154:816381:818575 [0] NCCL INFO + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8) n136-128-154:816382:818576 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 === n136-128-154:816381:818575 [0] NCCL INFO + PCI[24.0] - GPU/0-c5000 (0) n136-128-154:816382:818576 [1] NCCL INFO CPU/0-1 (1/1/2) n136-128-154:816381:818575 [0] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:816382:818576 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff) n136-128-154:816381:818575 [0] NCCL INFO + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8) n136-128-154:816382:818576 [1] NCCL INFO + PCI[24.0] - NIC/0-95000 n136-128-154:816381:818575 [0] NCCL INFO + PCI[24.0] - GPU/0-c9000 (1) n136-128-154:816382:818576 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff) n136-128-154:816381:818575 [0] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:816382:818576 [1] NCCL INFO + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8) n136-128-154:816381:818575 [0] NCCL INFO + PCI[24.0] - NIC/0-cc000 n136-128-154:816382:818576 [1] NCCL INFO + PCI[24.0] - GPU/0-c5000 (0) n136-128-154:816381:818575 [0] NCCL INFO + SYS[10.0] - CPU/0-0 n136-128-154:816382:818576 [1] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:816381:818575 [0] NCCL INFO CPU/0-0 (1/1/2) n136-128-154:816382:818576 [1] NCCL INFO + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8) n136-128-154:816381:818575 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff) n136-128-154:816382:818576 [1] NCCL INFO + PCI[24.0] - GPU/0-c9000 (1) n136-128-154:816381:818575 [0] NCCL INFO + PCI[24.0] - NIC/0-1d000 n136-128-154:816382:818576 [1] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:816381:818575 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff) n136-128-154:816382:818576 [1] NCCL INFO + PCI[24.0] - NIC/0-cc000 n136-128-154:816381:818575 [0] NCCL INFO + PCI[24.0] - NIC/0-60000 n136-128-154:816382:818576 [1] NCCL INFO + SYS[10.0] - CPU/0-0 n136-128-154:816381:818575 [0] NCCL INFO + SYS[10.0] - CPU/0-1 n136-128-154:816382:818576 [1] NCCL INFO CPU/0-0 (1/1/2) n136-128-154:816381:818575 [0] NCCL INFO ========================================== n136-128-154:816382:818576 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff) n136-128-154:816382:818576 [1] NCCL INFO + PCI[24.0] - NIC/0-1d000 n136-128-154:816382:818576 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff) n136-128-154:816381:818575 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:816382:818576 [1] NCCL INFO + PCI[24.0] - NIC/0-60000 n136-128-154:816382:818576 [1] NCCL INFO + SYS[10.0] - CPU/0-1 n136-128-154:816381:818575 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:816382:818576 [1] NCCL INFO ========================================== n136-128-154:816382:818576 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:816381:818575 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126 n136-128-154:816382:818576 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:816382:818576 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126 n136-128-154:816381:818575 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1 n136-128-154:816381:818575 [0] NCCL INFO 0 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816381:818575 [0] NCCL INFO 1 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816381:818575 [0] NCCL INFO 2 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816381:818575 [0] NCCL INFO 3 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816381:818575 [0] NCCL INFO 4 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816381:818575 [0] NCCL INFO 5 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816381:818575 [0] NCCL INFO 6 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816381:818575 [0] NCCL INFO 7 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816381:818575 [0] NCCL INFO 8 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816381:818575 [0] NCCL INFO 9 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816381:818575 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816381:818575 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816381:818575 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0 n136-128-154:816381:818575 [0] NCCL INFO 0 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816381:818575 [0] NCCL INFO 1 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816381:818575 [0] NCCL INFO 2 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816382:818576 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1 n136-128-154:816381:818575 [0] NCCL INFO 3 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816381:818575 [0] NCCL INFO 4 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816382:818576 [1] NCCL INFO 0 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816381:818575 [0] NCCL INFO 5 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816382:818576 [1] NCCL INFO 1 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816381:818575 [0] NCCL INFO 6 : GPU/0-c9000 GPU/0-c5000 n136-128-154:816382:818576 [1] NCCL INFO 2 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816381:818575 [0] NCCL INFO 7 : GPU/0-c9000 GPU/0-c5000 n136-128-154:816382:818576 [1] NCCL INFO 3 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816381:818575 [0] NCCL INFO 8 : GPU/0-c9000 GPU/0-c5000 n136-128-154:816382:818576 [1] NCCL INFO 4 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816381:818575 [0] NCCL INFO 9 : GPU/0-c9000 GPU/0-c5000 n136-128-154:816382:818576 [1] NCCL INFO 5 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816381:818575 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000 n136-128-154:816382:818576 [1] NCCL INFO 6 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816381:818575 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000 n136-128-154:816382:818576 [1] NCCL INFO 7 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816382:818576 [1] NCCL INFO 8 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816382:818576 [1] NCCL INFO 9 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816382:818576 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816382:818576 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816382:818576 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0 n136-128-154:816382:818576 [1] NCCL INFO 0 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816382:818576 [1] NCCL INFO 1 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816382:818576 [1] NCCL INFO 2 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816382:818576 [1] NCCL INFO 3 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816382:818576 [1] NCCL INFO 4 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816382:818576 [1] NCCL INFO 5 : GPU/0-c5000 GPU/0-c9000 n136-128-154:816382:818576 [1] NCCL INFO 6 : GPU/0-c9000 GPU/0-c5000 n136-128-154:816382:818576 [1] NCCL INFO 7 : GPU/0-c9000 GPU/0-c5000 n136-128-154:816382:818576 [1] NCCL INFO 8 : GPU/0-c9000 GPU/0-c5000 n136-128-154:816382:818576 [1] NCCL INFO 9 : GPU/0-c9000 GPU/0-c5000 n136-128-154:816382:818576 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000 n136-128-154:816382:818576 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000 n136-128-154:816381:818575 [0] NCCL INFO comm 0x11c77940 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0 n136-128-154:816382:818576 [1] NCCL INFO comm 0x11a96e00 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0 n136-128-154:816381:818575 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1 n136-128-154:816382:818576 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1 n136-128-154:816381:818575 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1 n136-128-154:816382:818576 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1 n136-128-154:816381:818575 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1 n136-128-154:816382:818576 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1 n136-128-154:816381:818575 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1 n136-128-154:816382:818576 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1 n136-128-154:816381:818575 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1 n136-128-154:816382:818576 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1 n136-128-154:816381:818575 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1 n136-128-154:816382:818576 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1 n136-128-154:816381:818575 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1 n136-128-154:816382:818576 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1 n136-128-154:816381:818575 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1 n136-128-154:816382:818576 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1 n136-128-154:816381:818575 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1 n136-128-154:816382:818576 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1 n136-128-154:816381:818575 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1 n136-128-154:816382:818576 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1 n136-128-154:816381:818575 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1 n136-128-154:816382:818576 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1 n136-128-154:816381:818575 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1 n136-128-154:816382:818576 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1 n136-128-154:816381:818575 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1 n136-128-154:816382:818576 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1 n136-128-154:816381:818575 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1 n136-128-154:816382:818576 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1 n136-128-154:816381:818575 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1 n136-128-154:816382:818576 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1 n136-128-154:816381:818575 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1 n136-128-154:816382:818576 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1 n136-128-154:816381:818575 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1 n136-128-154:816382:818576 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1 n136-128-154:816381:818575 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1 n136-128-154:816382:818576 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1 n136-128-154:816381:818575 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1 n136-128-154:816382:818576 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1 n136-128-154:816381:818575 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1 n136-128-154:816382:818576 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1 n136-128-154:816381:818575 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1 n136-128-154:816382:818576 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1 n136-128-154:816381:818575 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1 n136-128-154:816382:818576 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1 n136-128-154:816381:818575 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1 n136-128-154:816382:818576 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1 n136-128-154:816381:818575 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1 n136-128-154:816382:818576 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1 n136-128-154:816381:818575 [0] NCCL INFO Channel 00/24 : 0 1 n136-128-154:816381:818575 [0] NCCL INFO Channel 01/24 : 0 1 n136-128-154:816381:818575 [0] NCCL INFO Channel 02/24 : 0 1 n136-128-154:816382:818576 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0 n136-128-154:816381:818575 [0] NCCL INFO Channel 03/24 : 0 1 n136-128-154:816382:818576 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0 n136-128-154:816381:818575 [0] NCCL INFO Channel 04/24 : 0 1 n136-128-154:816382:818576 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0 n136-128-154:816381:818575 [0] NCCL INFO Channel 05/24 : 0 1 n136-128-154:816382:818576 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0 n136-128-154:816381:818575 [0] NCCL INFO Channel 06/24 : 0 1 n136-128-154:816382:818576 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0 n136-128-154:816381:818575 [0] NCCL INFO Channel 07/24 : 0 1 n136-128-154:816382:818576 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0 n136-128-154:816381:818575 [0] NCCL INFO Channel 08/24 : 0 1 n136-128-154:816382:818576 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0 n136-128-154:816381:818575 [0] NCCL INFO Channel 09/24 : 0 1 n136-128-154:816382:818576 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0 n136-128-154:816381:818575 [0] NCCL INFO Channel 10/24 : 0 1 n136-128-154:816382:818576 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0 n136-128-154:816381:818575 [0] NCCL INFO Channel 11/24 : 0 1 n136-128-154:816382:818576 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0 n136-128-154:816381:818575 [0] NCCL INFO Channel 12/24 : 0 1 n136-128-154:816382:818576 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0 n136-128-154:816381:818575 [0] NCCL INFO Channel 13/24 : 0 1 n136-128-154:816382:818576 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0 n136-128-154:816381:818575 [0] NCCL INFO Channel 14/24 : 0 1 n136-128-154:816382:818576 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0 n136-128-154:816381:818575 [0] NCCL INFO Channel 15/24 : 0 1 n136-128-154:816382:818576 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0 n136-128-154:816381:818575 [0] NCCL INFO Channel 16/24 : 0 1 n136-128-154:816382:818576 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0 n136-128-154:816381:818575 [0] NCCL INFO Channel 17/24 : 0 1 n136-128-154:816382:818576 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0 n136-128-154:816381:818575 [0] NCCL INFO Channel 18/24 : 0 1 n136-128-154:816382:818576 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0 n136-128-154:816381:818575 [0] NCCL INFO Channel 19/24 : 0 1 n136-128-154:816382:818576 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0 n136-128-154:816381:818575 [0] NCCL INFO Channel 20/24 : 0 1 n136-128-154:816382:818576 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0 n136-128-154:816381:818575 [0] NCCL INFO Channel 21/24 : 0 1 n136-128-154:816382:818576 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0 n136-128-154:816381:818575 [0] NCCL INFO Channel 22/24 : 0 1 n136-128-154:816382:818576 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0 n136-128-154:816381:818575 [0] NCCL INFO Channel 23/24 : 0 1 n136-128-154:816382:818576 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0 n136-128-154:816381:818575 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1 n136-128-154:816382:818576 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0 n136-128-154:816381:818575 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1 n136-128-154:816382:818576 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0 n136-128-154:816381:818575 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1 n136-128-154:816382:818576 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1 n136-128-154:816381:818575 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1 n136-128-154:816381:818575 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1 n136-128-154:816382:818576 [1] NCCL INFO P2P Chunksize set to 524288 n136-128-154:816381:818575 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1 n136-128-154:816381:818575 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1 n136-128-154:816381:818575 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1 n136-128-154:816381:818575 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1 n136-128-154:816381:818575 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1 n136-128-154:816381:818575 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1 n136-128-154:816381:818575 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1 n136-128-154:816381:818575 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1 n136-128-154:816381:818575 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1 n136-128-154:816381:818575 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1 n136-128-154:816381:818575 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1 n136-128-154:816381:818575 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1 n136-128-154:816381:818575 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1 n136-128-154:816381:818575 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1 n136-128-154:816381:818575 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1 n136-128-154:816381:818575 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1 n136-128-154:816381:818575 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1 n136-128-154:816381:818575 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1 n136-128-154:816381:818575 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1 n136-128-154:816381:818575 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1 n136-128-154:816381:818575 [0] NCCL INFO P2P Chunksize set to 524288 n136-128-154:816381:818575 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. n136-128-154:816381:818575 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 n136-128-154:816381:818588 [0] NCCL INFO [Proxy Service] Device 0 CPU core 34 n136-128-154:816381:818589 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 35 n136-128-154:816382:818576 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. n136-128-154:816382:818590 [1] NCCL INFO [Proxy Service] Device 1 CPU core 39 n136-128-154:816382:818591 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 40 n136-128-154:816381:818575 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 n136-128-154:816381:818575 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer n136-128-154:816381:818575 [0] NCCL INFO CC Off, workFifoBytes 1048576 n136-128-154:816382:818576 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 n136-128-154:816382:818576 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer n136-128-154:816382:818576 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. n136-128-154:816382:818576 [1] NCCL INFO ncclCommInitRankConfig comm 0x11a96e00 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x14739c438e31f4a1 - Init COMPLETE n136-128-154:816382:818576 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.77 (kernels 0.14, alloc 0.40, bootstrap 0.00, allgathers 0.00, topo 0.08, graphs 0.00, connections 0.01, rest 0.13) n136-128-154:816381:818575 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. n136-128-154:816381:818575 [0] NCCL INFO ncclCommInitRankConfig comm 0x11c77940 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x14739c438e31f4a1 - Init COMPLETE n136-128-154:816381:818575 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.78 (kernels 0.14, alloc 0.41, bootstrap 0.00, allgathers 0.00, topo 0.08, graphs 0.00, connections 0.01, rest 0.13) n136-128-154:816381:818592 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:816381:818592 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:816381:818592 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:816381:818592 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:816381:818592 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:816381:818592 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:816381:818592 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:816381:818592 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816381:818592 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816381:818592 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816381:818592 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816381:818592 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816381:818592 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816381:818592 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816381:818592 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816381:818592 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816381:818592 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816381:818592 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816381:818592 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816381:818592 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816381:818592 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816381:818592 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816381:818592 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816381:818592 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816382:818593 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 n136-128-154:816381:818592 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 2025-12-01:22:12:51 INFO [api.task:434] Building contexts for mmlu_anatomy on rank 0... 2025-12-01:22:12:51 INFO [api.task:434] Building contexts for mmlu_anatomy on rank 1... 0%| | 0/67 [00:00 0[6] via P2P/CUMEM/read n136-128-154:816382:820579 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:816382:820579 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read [2025-12-01 22:42:36] n136-128-154:816382:818590 [1] include/alloc.h:228 NCCL WARN Cuda failure 2 'out of memory' n136-128-154:816382:818590 [1] NCCL INFO transport/p2p.cc:217 -> 1 n136-128-154:816382:818590 [1] NCCL INFO transport/p2p.cc:650 -> 1 n136-128-154:816382:820579 [1] NCCL INFO transport/p2p.cc:421 -> 1 n136-128-154:816382:820579 [1] NCCL INFO transport.cc:35 -> 1 n136-128-154:816382:820579 [1] NCCL INFO transport.cc:156 -> 1 n136-128-154:816382:820579 [1] NCCL INFO group.cc:131 -> 1 n136-128-154:816382:820579 [1] NCCL INFO group.cc:73 -> 1 [Async thread] n136-128-154:816382:816382 [1] NCCL INFO group.cc:475 -> 1 n136-128-154:816382:816382 [1] NCCL INFO group.cc:694 -> 1 n136-128-154:816382:816382 [1] NCCL INFO group.cc:104 -> 1 [rank1]: Traceback (most recent call last): [rank1]: File "", line 198, in _run_module_as_main [rank1]: File "", line 88, in _run_code [rank1]: File "/mnt/bn/life-mllm/users/cxr/quantization/lm-evaluation-harness/lm_eval/__main__.py", line 530, in [rank1]: cli_evaluate() [rank1]: File "/mnt/bn/life-mllm/users/cxr/quantization/lm-evaluation-harness/lm_eval/__main__.py", line 449, in cli_evaluate [rank1]: results = evaluator.simple_evaluate( [rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank1]: File "/mnt/bn/life-mllm/users/cxr/quantization/lm-evaluation-harness/lm_eval/utils.py", line 439, in _wrapper [rank1]: return fn(*args, **kwargs) [rank1]: ^^^^^^^^^^^^^^^^^^^ [rank1]: File "/mnt/bn/life-mllm/users/cxr/quantization/lm-evaluation-harness/lm_eval/evaluator.py", line 342, in simple_evaluate [rank1]: results = evaluate( [rank1]: ^^^^^^^^^ [rank1]: File "/mnt/bn/life-mllm/users/cxr/quantization/lm-evaluation-harness/lm_eval/utils.py", line 439, in _wrapper [rank1]: return fn(*args, **kwargs) [rank1]: ^^^^^^^^^^^^^^^^^^^ [rank1]: File "/mnt/bn/life-mllm/users/cxr/quantization/lm-evaluation-harness/lm_eval/evaluator.py", line 669, in evaluate [rank1]: torch.distributed.gather_object( [rank1]: File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper [rank1]: return func(*args, **kwargs) [rank1]: ^^^^^^^^^^^^^^^^^^^^^ [rank1]: File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 3312, in gather_object [rank1]: gather( [rank1]: File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper [rank1]: return func(*args, **kwargs) [rank1]: ^^^^^^^^^^^^^^^^^^^^^ [rank1]: File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 4264, in gather [rank1]: work = group.gather(output_tensors, input_tensors, opts) [rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank1]: RuntimeError: NCCL Error 1: unhandled cuda error (run with NCCL_DEBUG=INFO for details) n136-128-154:816382:820583 [1] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:816382:820583 [1] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:816382:820583 [1] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:816382:820583 [1] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:816382:820583 [1] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:816382:820583 [1] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:816382:818590 [1] NCCL INFO misc/socket.cc:915 -> 3 [rank1]:[E1201 22:53:24.960908161 ProcessGroupNCCL.cpp:1858] [PG ID 0 PG GUID 0(default_pg) Rank 1] ProcessGroupNCCL's watchdog got stuck for 480 seconds without making progress in monitoring enqueued collectives. This typically indicates a NCCL/CUDA API (e.g., CudaEventDestroy) hang blocking the watchdog, and could be triggered by another thread holding the GIL inside a CUDA api (for example, CudaEventDestroy), or other deadlock-prone behaviors.If you suspect the watchdog is not actually stuck and a longer timeout would help, you can either increase the timeout (TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC) to a larger value or disable the heartbeat monitor (TORCH_NCCL_ENABLE_MONITORING=0).If either of aforementioned helps, feel free to file an issue to PyTorch about the short timeout or false positive abort; otherwise, please attempt to debug the hang. [rank1]:[E1201 22:53:24.962334558 ProcessGroupNCCL.cpp:1575] [PG ID 0 PG GUID 0(default_pg) Rank 1] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1 [rank1]:[F1201 23:01:24.969849995 ProcessGroupNCCL.cpp:1600] [PG ID 0 PG GUID 0(default_pg) Rank 1] [PG ID 0 PG GUID 0(default_pg) Rank 1] Terminating the process after attempting to dump debug info, due to ProcessGroupNCCL watchdog hang. W1201 23:01:31.658000 816337 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 816381 closing signal SIGTERM E1201 23:01:32.177000 816337 site-packages/torch/distributed/elastic/multiprocessing/api.py:882] failed (exitcode: -6) local_rank: 1 (pid: 816382) of binary: /mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/bin/python Traceback (most recent call last): File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/bin/accelerate", line 7, in sys.exit(main()) ^^^^^^ File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 50, in main args.func(args) File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/accelerate/commands/launch.py", line 1272, in launch_command multi_gpu_launcher(args) File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/accelerate/commands/launch.py", line 899, in multi_gpu_launcher distrib_run.run(args) File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/torch/distributed/run.py", line 927, in run elastic_launch( File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 156, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/bn/life-mllm/users/cxr/zhy/miniconda3/envs/lm-evaluation-harness/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 293, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ======================================================= lm_eval FAILED ------------------------------------------------------- Failures: ------------------------------------------------------- Root Cause (first observed failure): [0]: time : 2025-12-01_23:01:31 host : n136-128-154.byted.org rank : 1 (local_rank: 1) exitcode : -6 (pid: 816382) error_file: traceback : Signal 6 (SIGABRT) received by PID 816382 ======================================================= 警告:任务 mmlu 执行失败! ================================================== 开始评估:任务=arc_challenge | 少样本数=25 | 模型=Qwen2.5-32B 输出路径:results2/Qwen2.5-32B/base_arc_challenge.json ================================================== The following values were not passed to `accelerate launch` and had defaults used instead: More than one GPU was found, enabling multi-GPU training. If this was unintended please pass in `--num_processes=1`. `--num_machines` was set to a value of `1` `--mixed_precision` was set to a value of `'no'` `--dynamo_backend` was set to a value of `'no'` To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`. 2025-12-01:23:02:22 INFO [__main__:440] Selected Tasks: ['arc_challenge'] 2025-12-01:23:02:22 INFO [__main__:440] Selected Tasks: ['arc_challenge'] 2025-12-01:23:02:22 INFO [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2025-12-01:23:02:22 INFO [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-32B'} 2025-12-01:23:02:22 INFO [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2025-12-01:23:02:22 INFO [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-32B'} 2025-12-01:23:02:23 WARNING [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. 2025-12-01:23:02:23 INFO [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'} 2025-12-01:23:02:23 INFO [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'} `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! Loading checkpoint shards: 0%| | 0/17 [00:00 n136-128-154:821893:822105 [0] NCCL INFO Initialized NET plugin IB n136-128-154:821893:822105 [0] NCCL INFO Assigned NET plugin IB to comm n136-128-154:821893:822105 [0] NCCL INFO Using network IB n136-128-154:821893:822105 [0] NCCL INFO ncclCommInitRankConfig comm 0x114968c0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x183565ff2a06145b - Init START 62%|██████▏ | 364/586 [00:09<00:06, 36.82it/s] 63%|██████▎ | 368/586 [00:10<00:05, 36.85it/s] 63%|██████▎ | 372/586 [00:10<00:05, 36.89it/s] 64%|██████▍ | 376/586 [00:10<00:05, 36.90it/s] 65%|██████▍ | 380/586 [00:10<00:05, 36.93it/s] 66%|██████▌ | 384/586 [00:10<00:05, 36.56it/s] 66%|██████▌ | 388/586 [00:10<00:05, 36.67it/s] 67%|██████▋ | 392/586 [00:10<00:05, 36.71it/s] 68%|██████▊ | 396/586 [00:10<00:05, 36.76it/s] 68%|██████▊ | 400/586 [00:10<00:05, 36.76it/s] 69%|██████▉ | 404/586 [00:11<00:04, 36.73it/s] 70%|██████▉ | 408/586 [00:11<00:04, 36.82it/s] 70%|███████ | 412/586 [00:11<00:04, 36.89it/s] 71%|███████ | 416/586 [00:11<00:04, 36.90it/s] 72%|███████▏ | 420/586 [00:11<00:04, 36.86it/s] 72%|███████▏ | 424/586 [00:11<00:04, 36.90it/s] 73%|███████▎ | 428/586 [00:11<00:04, 36.89it/s] 74%|███████▎ | 432/586 [00:11<00:04, 36.90it/s] 74%|███████▍ | 436/586 [00:11<00:04, 36.96it/s] 75%|███████▌ | 440/586 [00:11<00:03, 36.99it/s] 76%|███████▌ | 444/586 [00:12<00:03, 36.98it/s] 76%|███████▋ | 448/586 [00:12<00:03, 37.00it/s] 77%|███████▋ | 452/586 [00:12<00:03, 37.01it/s] 78%|███████▊ | 456/586 [00:12<00:03, 37.04it/s] 78%|███████▊ | 460/586 [00:12<00:03, 37.09it/s] 79%|███████▉ | 464/586 [00:12<00:03, 36.72it/s] 80%|███████▉ | 468/586 [00:12<00:03, 36.83it/s] 81%|████████ | 472/586 [00:12<00:03, 36.93it/s] 81%|████████ | 476/586 [00:12<00:02, 36.89it/s] 82%|████████▏ | 480/586 [00:13<00:02, 36.80it/s] 83%|████████▎ | 484/586 [00:13<00:02, 36.88it/s] 83%|████████▎ | 488/586 [00:13<00:02, 36.94it/s] 84%|████████▍ | 492/586 [00:13<00:02, 36.99it/s] 85%|████████▍ | 496/586 [00:13<00:02, 37.00it/s] 85%|████████▌ | 500/586 [00:13<00:02, 37.05it/s] 86%|████████▌ | 504/586 [00:13<00:02, 36.67it/s] 87%|████████▋ | 508/586 [00:13<00:02, 36.75it/s] 87%|████████▋ | 512/586 [00:13<00:02, 36.84it/s] 88%|████████▊ | 516/586 [00:14<00:01, 36.95it/s] 89%|████████▊ | 520/586 [00:14<00:01, 36.99it/s] 89%|████████▉ | 524/586 [00:14<00:01, 37.01it/s] 90%|█████████ | 528/586 [00:14<00:01, 37.08it/s] 91%|█████████ | 532/586 [00:14<00:01, 37.11it/s] 91%|█████████▏| 536/586 [00:14<00:01, 37.12it/s] 92%|█████████▏| 540/586 [00:14<00:01, 37.08it/s] 93%|█████████▎| 544/586 [00:14<00:01, 37.09it/s] 94%|█████████▎| 548/586 [00:14<00:01, 37.02it/s] 94%|█████████▍| 552/586 [00:15<00:00, 37.06it/s] 95%|█████████▍| 556/586 [00:15<00:00, 37.11it/s] 96%|█████████▌| 560/586 [00:15<00:00, 36.97it/s] 96%|█████████▌| 564/586 [00:15<00:00, 36.86it/s] 97%|█████████▋| 568/586 [00:15<00:00, 36.61it/s] 98%|█████████▊| 572/586 [00:15<00:00, 36.44it/s] 98%|█████████▊| 576/586 [00:15<00:00, 36.46it/s] 99%|█████████▉| 580/586 [00:15<00:00, 36.45it/s] 100%|█████████▉| 584/586 [00:15<00:00, 36.70it/s] 100%|██████████| 586/586 [00:15<00:00, 36.78it/s] n136-128-154:821894:821894 [1] NCCL INFO cudaDriverVersion 12040 n136-128-154:821894:821894 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6 n136-128-154:821894:821894 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0 n136-128-154:821894:821894 [1] NCCL INFO NCCL version 2.27.5+cuda12.9 n136-128-154:821894:821894 [1] NCCL INFO Comm config Blocking set to 1 n136-128-154:821894:822139 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. n136-128-154:821894:822139 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0. n136-128-154:821894:822139 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6 n136-128-154:821894:822139 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0 n136-128-154:821894:822139 [1] NCCL INFO NCCL_IB_HCA set to mlx5 n136-128-154:821894:822139 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. n136-128-154:821894:822139 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0> n136-128-154:821894:822139 [1] NCCL INFO Initialized NET plugin IB n136-128-154:821894:822139 [1] NCCL INFO Assigned NET plugin IB to comm n136-128-154:821894:822139 [1] NCCL INFO Using network IB n136-128-154:821894:822139 [1] NCCL INFO ncclCommInitRankConfig comm 0x113552a0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x183565ff2a06145b - Init START n136-128-154:821894:822139 [1] NCCL INFO RAS client listening socket at ::1<28028> n136-128-154:821893:822105 [0] NCCL INFO RAS client listening socket at ::1<28028> n136-128-154:821894:822139 [1] NCCL INFO TOPO/NET : Importing network plugins to topology n136-128-154:821894:822139 [1] NCCL INFO Retrieving state for IB n136-128-154:821894:822139 [1] NCCL INFO Initialized state 0 for IB n136-128-154:821894:822139 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null) n136-128-154:821894:822139 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null) n136-128-154:821894:822139 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null) n136-128-154:821894:822139 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null) n136-128-154:821893:822105 [0] NCCL INFO TOPO/NET : Importing network plugins to topology n136-128-154:821893:822105 [0] NCCL INFO Retrieving state for IB n136-128-154:821893:822105 [0] NCCL INFO Initialized state 0 for IB n136-128-154:821893:822105 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null) n136-128-154:821893:822105 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null) n136-128-154:821893:822105 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null) n136-128-154:821893:822105 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null) n136-128-154:821894:822139 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default n136-128-154:821894:822139 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default n136-128-154:821894:822139 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 === n136-128-154:821894:822139 [1] NCCL INFO CPU/0-1 (1/1/2) n136-128-154:821894:822139 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff) n136-128-154:821894:822139 [1] NCCL INFO + PCI[24.0] - NIC/0-95000 n136-128-154:821894:822139 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff) n136-128-154:821894:822139 [1] NCCL INFO + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8) n136-128-154:821894:822139 [1] NCCL INFO + PCI[24.0] - GPU/0-c5000 (0) n136-128-154:821894:822139 [1] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:821894:822139 [1] NCCL INFO + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8) n136-128-154:821894:822139 [1] NCCL INFO + PCI[24.0] - GPU/0-c9000 (1) n136-128-154:821894:822139 [1] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:821894:822139 [1] NCCL INFO + PCI[24.0] - NIC/0-cc000 n136-128-154:821894:822139 [1] NCCL INFO + SYS[10.0] - CPU/0-0 n136-128-154:821894:822139 [1] NCCL INFO CPU/0-0 (1/1/2) n136-128-154:821894:822139 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff) n136-128-154:821894:822139 [1] NCCL INFO + PCI[24.0] - NIC/0-1d000 n136-128-154:821894:822139 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff) n136-128-154:821894:822139 [1] NCCL INFO + PCI[24.0] - NIC/0-60000 n136-128-154:821894:822139 [1] NCCL INFO + SYS[10.0] - CPU/0-1 n136-128-154:821894:822139 [1] NCCL INFO ========================================== n136-128-154:821894:822139 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:821894:822139 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:821894:822139 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126 n136-128-154:821894:822139 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1 n136-128-154:821894:822139 [1] NCCL INFO 0 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821894:822139 [1] NCCL INFO 1 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821894:822139 [1] NCCL INFO 2 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821894:822139 [1] NCCL INFO 3 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821894:822139 [1] NCCL INFO 4 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821894:822139 [1] NCCL INFO 5 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821894:822139 [1] NCCL INFO 6 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821894:822139 [1] NCCL INFO 7 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821894:822139 [1] NCCL INFO 8 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821894:822139 [1] NCCL INFO 9 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821894:822139 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821894:822139 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821894:822139 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0 n136-128-154:821894:822139 [1] NCCL INFO 0 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821894:822139 [1] NCCL INFO 1 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821894:822139 [1] NCCL INFO 2 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821894:822139 [1] NCCL INFO 3 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821894:822139 [1] NCCL INFO 4 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821894:822139 [1] NCCL INFO 5 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821894:822139 [1] NCCL INFO 6 : GPU/0-c9000 GPU/0-c5000 n136-128-154:821894:822139 [1] NCCL INFO 7 : GPU/0-c9000 GPU/0-c5000 n136-128-154:821894:822139 [1] NCCL INFO 8 : GPU/0-c9000 GPU/0-c5000 n136-128-154:821894:822139 [1] NCCL INFO 9 : GPU/0-c9000 GPU/0-c5000 n136-128-154:821894:822139 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000 n136-128-154:821894:822139 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000 n136-128-154:821893:822105 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default n136-128-154:821893:822105 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default n136-128-154:821893:822105 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 === n136-128-154:821893:822105 [0] NCCL INFO CPU/0-1 (1/1/2) n136-128-154:821893:822105 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff) n136-128-154:821893:822105 [0] NCCL INFO + PCI[24.0] - NIC/0-95000 n136-128-154:821893:822105 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff) n136-128-154:821893:822105 [0] NCCL INFO + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8) n136-128-154:821893:822105 [0] NCCL INFO + PCI[24.0] - GPU/0-c5000 (0) n136-128-154:821893:822105 [0] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:821893:822105 [0] NCCL INFO + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8) n136-128-154:821893:822105 [0] NCCL INFO + PCI[24.0] - GPU/0-c9000 (1) n136-128-154:821893:822105 [0] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:821893:822105 [0] NCCL INFO + PCI[24.0] - NIC/0-cc000 n136-128-154:821893:822105 [0] NCCL INFO + SYS[10.0] - CPU/0-0 n136-128-154:821893:822105 [0] NCCL INFO CPU/0-0 (1/1/2) n136-128-154:821893:822105 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff) n136-128-154:821893:822105 [0] NCCL INFO + PCI[24.0] - NIC/0-1d000 n136-128-154:821893:822105 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff) n136-128-154:821893:822105 [0] NCCL INFO + PCI[24.0] - NIC/0-60000 n136-128-154:821893:822105 [0] NCCL INFO + SYS[10.0] - CPU/0-1 n136-128-154:821893:822105 [0] NCCL INFO ========================================== n136-128-154:821893:822105 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:821893:822105 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:821893:822105 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126 n136-128-154:821893:822105 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1 n136-128-154:821893:822105 [0] NCCL INFO 0 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821893:822105 [0] NCCL INFO 1 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821893:822105 [0] NCCL INFO 2 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821893:822105 [0] NCCL INFO 3 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821893:822105 [0] NCCL INFO 4 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821893:822105 [0] NCCL INFO 5 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821893:822105 [0] NCCL INFO 6 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821893:822105 [0] NCCL INFO 7 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821893:822105 [0] NCCL INFO 8 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821893:822105 [0] NCCL INFO 9 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821893:822105 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821893:822105 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821893:822105 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0 n136-128-154:821893:822105 [0] NCCL INFO 0 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821893:822105 [0] NCCL INFO 1 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821893:822105 [0] NCCL INFO 2 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821893:822105 [0] NCCL INFO 3 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821893:822105 [0] NCCL INFO 4 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821893:822105 [0] NCCL INFO 5 : GPU/0-c5000 GPU/0-c9000 n136-128-154:821893:822105 [0] NCCL INFO 6 : GPU/0-c9000 GPU/0-c5000 n136-128-154:821893:822105 [0] NCCL INFO 7 : GPU/0-c9000 GPU/0-c5000 n136-128-154:821893:822105 [0] NCCL INFO 8 : GPU/0-c9000 GPU/0-c5000 n136-128-154:821893:822105 [0] NCCL INFO 9 : GPU/0-c9000 GPU/0-c5000 n136-128-154:821893:822105 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000 n136-128-154:821893:822105 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000 n136-128-154:821894:822139 [1] NCCL INFO comm 0x113552a0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0 n136-128-154:821893:822105 [0] NCCL INFO comm 0x114968c0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0 n136-128-154:821894:822139 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1 n136-128-154:821894:822139 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1 n136-128-154:821894:822139 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1 n136-128-154:821894:822139 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1 n136-128-154:821894:822139 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1 n136-128-154:821894:822139 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1 n136-128-154:821894:822139 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1 n136-128-154:821894:822139 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1 n136-128-154:821894:822139 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1 n136-128-154:821894:822139 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1 n136-128-154:821894:822139 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1 n136-128-154:821894:822139 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1 n136-128-154:821894:822139 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1 n136-128-154:821894:822139 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1 n136-128-154:821894:822139 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1 n136-128-154:821894:822139 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1 n136-128-154:821894:822139 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1 n136-128-154:821894:822139 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1 n136-128-154:821894:822139 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1 n136-128-154:821894:822139 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1 n136-128-154:821894:822139 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1 n136-128-154:821894:822139 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1 n136-128-154:821894:822139 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1 n136-128-154:821894:822139 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1 n136-128-154:821893:822105 [0] NCCL INFO Channel 00/24 : 0 1 n136-128-154:821893:822105 [0] NCCL INFO Channel 01/24 : 0 1 n136-128-154:821894:822139 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0 n136-128-154:821893:822105 [0] NCCL INFO Channel 02/24 : 0 1 n136-128-154:821894:822139 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0 n136-128-154:821893:822105 [0] NCCL INFO Channel 03/24 : 0 1 n136-128-154:821894:822139 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0 n136-128-154:821893:822105 [0] NCCL INFO Channel 04/24 : 0 1 n136-128-154:821894:822139 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0 n136-128-154:821893:822105 [0] NCCL INFO Channel 05/24 : 0 1 n136-128-154:821894:822139 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0 n136-128-154:821893:822105 [0] NCCL INFO Channel 06/24 : 0 1 n136-128-154:821894:822139 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0 n136-128-154:821893:822105 [0] NCCL INFO Channel 07/24 : 0 1 n136-128-154:821894:822139 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0 n136-128-154:821893:822105 [0] NCCL INFO Channel 08/24 : 0 1 n136-128-154:821894:822139 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0 n136-128-154:821893:822105 [0] NCCL INFO Channel 09/24 : 0 1 n136-128-154:821894:822139 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0 n136-128-154:821893:822105 [0] NCCL INFO Channel 10/24 : 0 1 n136-128-154:821894:822139 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0 n136-128-154:821893:822105 [0] NCCL INFO Channel 11/24 : 0 1 n136-128-154:821894:822139 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0 n136-128-154:821893:822105 [0] NCCL INFO Channel 12/24 : 0 1 n136-128-154:821894:822139 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0 n136-128-154:821893:822105 [0] NCCL INFO Channel 13/24 : 0 1 n136-128-154:821894:822139 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0 n136-128-154:821893:822105 [0] NCCL INFO Channel 14/24 : 0 1 n136-128-154:821894:822139 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0 n136-128-154:821893:822105 [0] NCCL INFO Channel 15/24 : 0 1 n136-128-154:821894:822139 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0 n136-128-154:821893:822105 [0] NCCL INFO Channel 16/24 : 0 1 n136-128-154:821894:822139 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0 n136-128-154:821893:822105 [0] NCCL INFO Channel 17/24 : 0 1 n136-128-154:821894:822139 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0 n136-128-154:821893:822105 [0] NCCL INFO Channel 18/24 : 0 1 n136-128-154:821894:822139 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0 n136-128-154:821893:822105 [0] NCCL INFO Channel 19/24 : 0 1 n136-128-154:821894:822139 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0 n136-128-154:821893:822105 [0] NCCL INFO Channel 20/24 : 0 1 n136-128-154:821894:822139 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0 n136-128-154:821893:822105 [0] NCCL INFO Channel 21/24 : 0 1 n136-128-154:821894:822139 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0 n136-128-154:821893:822105 [0] NCCL INFO Channel 22/24 : 0 1 n136-128-154:821894:822139 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0 n136-128-154:821893:822105 [0] NCCL INFO Channel 23/24 : 0 1 n136-128-154:821894:822139 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0 n136-128-154:821894:822139 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0 n136-128-154:821893:822105 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1 n136-128-154:821894:822139 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1 n136-128-154:821893:822105 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1 n136-128-154:821893:822105 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1 n136-128-154:821894:822139 [1] NCCL INFO P2P Chunksize set to 524288 n136-128-154:821893:822105 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1 n136-128-154:821893:822105 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1 n136-128-154:821893:822105 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1 n136-128-154:821893:822105 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1 n136-128-154:821893:822105 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1 n136-128-154:821893:822105 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1 n136-128-154:821893:822105 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1 n136-128-154:821893:822105 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1 n136-128-154:821893:822105 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1 n136-128-154:821893:822105 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1 n136-128-154:821893:822105 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1 n136-128-154:821893:822105 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1 n136-128-154:821893:822105 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1 n136-128-154:821893:822105 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1 n136-128-154:821893:822105 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1 n136-128-154:821893:822105 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1 n136-128-154:821893:822105 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1 n136-128-154:821893:822105 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1 n136-128-154:821893:822105 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1 n136-128-154:821893:822105 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1 n136-128-154:821893:822105 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1 n136-128-154:821893:822105 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1 n136-128-154:821893:822105 [0] NCCL INFO P2P Chunksize set to 524288 n136-128-154:821894:822139 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. n136-128-154:821894:822146 [1] NCCL INFO [Proxy Service] Device 1 CPU core 112 n136-128-154:821894:822147 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 49 n136-128-154:821893:822105 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. n136-128-154:821893:822105 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 n136-128-154:821893:822148 [0] NCCL INFO [Proxy Service] Device 0 CPU core 36 n136-128-154:821893:822149 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 37 n136-128-154:821894:822139 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 n136-128-154:821894:822139 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer n136-128-154:821893:822105 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 n136-128-154:821893:822105 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer n136-128-154:821893:822105 [0] NCCL INFO CC Off, workFifoBytes 1048576 n136-128-154:821894:822139 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. n136-128-154:821894:822139 [1] NCCL INFO ncclCommInitRankConfig comm 0x113552a0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x183565ff2a06145b - Init COMPLETE n136-128-154:821894:822139 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.54 (kernels 0.12, alloc 0.23, bootstrap 0.00, allgathers 0.01, topo 0.08, graphs 0.00, connections 0.01, rest 0.09) n136-128-154:821893:822105 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. n136-128-154:821893:822105 [0] NCCL INFO ncclCommInitRankConfig comm 0x114968c0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x183565ff2a06145b - Init COMPLETE n136-128-154:821893:822105 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 6.96 (kernels 0.15, alloc 0.23, bootstrap 6.38, allgathers 0.01, topo 0.08, graphs 0.00, connections 0.01, rest 0.09) n136-128-154:821894:822150 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822150 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822150 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822150 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822150 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822150 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821893:822151 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:821894:822150 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821893:822151 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:821894:822150 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821893:822151 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:821894:822150 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821893:822151 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:821894:822150 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821893:822151 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:821894:822150 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821893:822151 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:821894:822150 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821893:822151 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:821894:822150 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821893:822151 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:821894:822150 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821893:822151 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:821894:822150 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821893:822151 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:821894:822150 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822150 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821893:822151 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:821894:822150 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821893:822151 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:821894:822150 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821893:822151 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:821894:822150 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821893:822151 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:821894:822150 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821893:822151 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:821894:822150 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821893:822151 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:821894:822150 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821893:822151 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:821894:822150 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821893:822151 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:821893:822151 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:821893:822151 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:821893:822151 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:821893:822151 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:821893:822151 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:821893:822151 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:821894:822150 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 n136-128-154:821893:822151 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 2025-12-01:23:03:21 INFO [evaluator:559] Running loglikelihood requests 2025-12-01:23:03:21 INFO [evaluator:559] Running loglikelihood requests Passed argument batch_size = auto:1. Detecting largest batch size Running loglikelihood requests: 0%| | 0/2344 [00:00 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:821894:822970 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization' To add an exception for this directory, call: git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization n136-128-154:821894:822974 [1] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:821894:822974 [1] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:821894:822974 [1] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:821894:822974 [1] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:821894:822974 [1] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:821894:822974 [1] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:821894:822146 [1] NCCL INFO misc/socket.cc:915 -> 3 2025-12-01:23:15:58 INFO [loggers.evaluation_tracker:209] Saving results aggregated hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-32B), gen_kwargs: (None), limit: None, num_fewshot: 25, batch_size: auto (9) | Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr| |-------------|------:|------|-----:|--------|---|-----:|---|-----:| |arc_challenge| 1|none | 25|acc |↑ |0.6655|± |0.0138| | | |none | 25|acc_norm|↑ |0.6945|± |0.0135| [rank0]:[W1201 23:15:58.985377956 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) n136-128-154:821893:823024 [0] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:821893:823024 [0] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:821893:823024 [0] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:821893:822148 [0] NCCL INFO misc/socket.cc:915 -> 3 n136-128-154:821893:823024 [0] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:821893:823024 [0] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:821893:823024 [0] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:821894:822146 [1] NCCL INFO misc/socket.cc:915 -> 3 n136-128-154:821894:822974 [1] NCCL INFO comm 0x113552a0 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE n136-128-154:821893:823024 [0] NCCL INFO comm 0x114968c0 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE 任务 arc_challenge 评估完成! ================================================== 开始评估:任务=truthfulqa_mc1 | 少样本数=0 | 模型=Qwen2.5-32B 输出路径:results2/Qwen2.5-32B/base_truthfulqa_mc1.json ================================================== The following values were not passed to `accelerate launch` and had defaults used instead: More than one GPU was found, enabling multi-GPU training. If this was unintended please pass in `--num_processes=1`. `--num_machines` was set to a value of `1` `--mixed_precision` was set to a value of `'no'` `--dynamo_backend` was set to a value of `'no'` To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`. 2025-12-01:23:16:48 INFO [__main__:440] Selected Tasks: ['truthfulqa_mc1'] 2025-12-01:23:16:48 INFO [__main__:440] Selected Tasks: ['truthfulqa_mc1'] 2025-12-01:23:16:48 INFO [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2025-12-01:23:16:48 INFO [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-32B'} 2025-12-01:23:16:48 INFO [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2025-12-01:23:16:48 INFO [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-32B'} 2025-12-01:23:16:49 WARNING [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. 2025-12-01:23:16:49 INFO [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'} `torch_dtype` is deprecated! Use `dtype` instead! 2025-12-01:23:16:49 INFO [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'} `torch_dtype` is deprecated! Use `dtype` instead! Loading checkpoint shards: 0%| | 0/17 [00:00 n136-128-154:823070:823256 [0] NCCL INFO Initialized NET plugin IB n136-128-154:823070:823256 [0] NCCL INFO Assigned NET plugin IB to comm n136-128-154:823070:823256 [0] NCCL INFO Using network IB n136-128-154:823070:823256 [0] NCCL INFO ncclCommInitRankConfig comm 0x11703740 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xc7f0e4b2f8d4055b - Init START 2025-12-01:23:17:25 INFO [evaluator:305] num_fewshot has been set to 0 for truthfulqa_mc1 in its config. Manual configuration will be ignored. 2025-12-01:23:17:25 INFO [api.task:434] Building contexts for truthfulqa_mc1 on rank 1... 0%| | 0/408 [00:00 n136-128-154:823071:823265 [1] NCCL INFO Initialized NET plugin IB n136-128-154:823071:823265 [1] NCCL INFO Assigned NET plugin IB to comm n136-128-154:823071:823265 [1] NCCL INFO Using network IB n136-128-154:823071:823265 [1] NCCL INFO ncclCommInitRankConfig comm 0x1199ce00 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xc7f0e4b2f8d4055b - Init START n136-128-154:823071:823265 [1] NCCL INFO RAS client listening socket at ::1<28028> n136-128-154:823070:823256 [0] NCCL INFO RAS client listening socket at ::1<28028> n136-128-154:823071:823265 [1] NCCL INFO TOPO/NET : Importing network plugins to topology n136-128-154:823071:823265 [1] NCCL INFO Retrieving state for IB n136-128-154:823071:823265 [1] NCCL INFO Initialized state 0 for IB n136-128-154:823071:823265 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null) n136-128-154:823070:823256 [0] NCCL INFO TOPO/NET : Importing network plugins to topology n136-128-154:823070:823256 [0] NCCL INFO Retrieving state for IB n136-128-154:823070:823256 [0] NCCL INFO Initialized state 0 for IB n136-128-154:823071:823265 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null) n136-128-154:823070:823256 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null) n136-128-154:823071:823265 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null) n136-128-154:823070:823256 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null) n136-128-154:823071:823265 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null) n136-128-154:823070:823256 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null) n136-128-154:823070:823256 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null) n136-128-154:823071:823265 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default n136-128-154:823070:823256 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default n136-128-154:823071:823265 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default n136-128-154:823070:823256 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default n136-128-154:823071:823265 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 === n136-128-154:823070:823256 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 === n136-128-154:823071:823265 [1] NCCL INFO CPU/0-1 (1/1/2) n136-128-154:823070:823256 [0] NCCL INFO CPU/0-1 (1/1/2) n136-128-154:823071:823265 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff) n136-128-154:823070:823256 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff) n136-128-154:823071:823265 [1] NCCL INFO + PCI[24.0] - NIC/0-95000 n136-128-154:823070:823256 [0] NCCL INFO + PCI[24.0] - NIC/0-95000 n136-128-154:823071:823265 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff) n136-128-154:823070:823256 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff) n136-128-154:823071:823265 [1] NCCL INFO + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8) n136-128-154:823070:823256 [0] NCCL INFO + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8) n136-128-154:823071:823265 [1] NCCL INFO + PCI[24.0] - GPU/0-c5000 (0) n136-128-154:823070:823256 [0] NCCL INFO + PCI[24.0] - GPU/0-c5000 (0) n136-128-154:823071:823265 [1] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:823070:823256 [0] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:823071:823265 [1] NCCL INFO + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8) n136-128-154:823070:823256 [0] NCCL INFO + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8) n136-128-154:823071:823265 [1] NCCL INFO + PCI[24.0] - GPU/0-c9000 (1) n136-128-154:823070:823256 [0] NCCL INFO + PCI[24.0] - GPU/0-c9000 (1) n136-128-154:823071:823265 [1] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:823070:823256 [0] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:823071:823265 [1] NCCL INFO + PCI[24.0] - NIC/0-cc000 n136-128-154:823070:823256 [0] NCCL INFO + PCI[24.0] - NIC/0-cc000 n136-128-154:823071:823265 [1] NCCL INFO + SYS[10.0] - CPU/0-0 n136-128-154:823070:823256 [0] NCCL INFO + SYS[10.0] - CPU/0-0 n136-128-154:823071:823265 [1] NCCL INFO CPU/0-0 (1/1/2) n136-128-154:823070:823256 [0] NCCL INFO CPU/0-0 (1/1/2) n136-128-154:823071:823265 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff) n136-128-154:823070:823256 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff) n136-128-154:823071:823265 [1] NCCL INFO + PCI[24.0] - NIC/0-1d000 n136-128-154:823070:823256 [0] NCCL INFO + PCI[24.0] - NIC/0-1d000 n136-128-154:823071:823265 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff) n136-128-154:823070:823256 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff) n136-128-154:823071:823265 [1] NCCL INFO + PCI[24.0] - NIC/0-60000 n136-128-154:823070:823256 [0] NCCL INFO + PCI[24.0] - NIC/0-60000 n136-128-154:823071:823265 [1] NCCL INFO + SYS[10.0] - CPU/0-1 n136-128-154:823070:823256 [0] NCCL INFO + SYS[10.0] - CPU/0-1 n136-128-154:823071:823265 [1] NCCL INFO ========================================== n136-128-154:823070:823256 [0] NCCL INFO ========================================== n136-128-154:823071:823265 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:823070:823256 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:823071:823265 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:823070:823256 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:823071:823265 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126 n136-128-154:823070:823256 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126 n136-128-154:823071:823265 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1 n136-128-154:823071:823265 [1] NCCL INFO 0 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823071:823265 [1] NCCL INFO 1 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823071:823265 [1] NCCL INFO 2 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823071:823265 [1] NCCL INFO 3 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823071:823265 [1] NCCL INFO 4 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823071:823265 [1] NCCL INFO 5 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823071:823265 [1] NCCL INFO 6 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823071:823265 [1] NCCL INFO 7 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823071:823265 [1] NCCL INFO 8 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823071:823265 [1] NCCL INFO 9 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823071:823265 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823071:823265 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823071:823265 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0 n136-128-154:823071:823265 [1] NCCL INFO 0 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823071:823265 [1] NCCL INFO 1 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823071:823265 [1] NCCL INFO 2 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823071:823265 [1] NCCL INFO 3 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823071:823265 [1] NCCL INFO 4 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823071:823265 [1] NCCL INFO 5 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823071:823265 [1] NCCL INFO 6 : GPU/0-c9000 GPU/0-c5000 n136-128-154:823071:823265 [1] NCCL INFO 7 : GPU/0-c9000 GPU/0-c5000 n136-128-154:823071:823265 [1] NCCL INFO 8 : GPU/0-c9000 GPU/0-c5000 n136-128-154:823071:823265 [1] NCCL INFO 9 : GPU/0-c9000 GPU/0-c5000 n136-128-154:823071:823265 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000 n136-128-154:823071:823265 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000 n136-128-154:823070:823256 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1 n136-128-154:823070:823256 [0] NCCL INFO 0 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823070:823256 [0] NCCL INFO 1 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823070:823256 [0] NCCL INFO 2 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823070:823256 [0] NCCL INFO 3 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823070:823256 [0] NCCL INFO 4 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823070:823256 [0] NCCL INFO 5 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823070:823256 [0] NCCL INFO 6 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823070:823256 [0] NCCL INFO 7 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823070:823256 [0] NCCL INFO 8 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823070:823256 [0] NCCL INFO 9 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823070:823256 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823070:823256 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823070:823256 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0 n136-128-154:823070:823256 [0] NCCL INFO 0 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823070:823256 [0] NCCL INFO 1 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823070:823256 [0] NCCL INFO 2 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823070:823256 [0] NCCL INFO 3 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823070:823256 [0] NCCL INFO 4 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823070:823256 [0] NCCL INFO 5 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823070:823256 [0] NCCL INFO 6 : GPU/0-c9000 GPU/0-c5000 n136-128-154:823070:823256 [0] NCCL INFO 7 : GPU/0-c9000 GPU/0-c5000 n136-128-154:823070:823256 [0] NCCL INFO 8 : GPU/0-c9000 GPU/0-c5000 n136-128-154:823070:823256 [0] NCCL INFO 9 : GPU/0-c9000 GPU/0-c5000 n136-128-154:823070:823256 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000 n136-128-154:823070:823256 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000 n136-128-154:823071:823265 [1] NCCL INFO comm 0x1199ce00 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0 n136-128-154:823070:823256 [0] NCCL INFO comm 0x11703740 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0 n136-128-154:823071:823265 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1 n136-128-154:823071:823265 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1 n136-128-154:823071:823265 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1 n136-128-154:823071:823265 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1 n136-128-154:823071:823265 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1 n136-128-154:823071:823265 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1 n136-128-154:823071:823265 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1 n136-128-154:823071:823265 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1 n136-128-154:823071:823265 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1 n136-128-154:823071:823265 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1 n136-128-154:823071:823265 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1 n136-128-154:823071:823265 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1 n136-128-154:823071:823265 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1 n136-128-154:823071:823265 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1 n136-128-154:823071:823265 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1 n136-128-154:823071:823265 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1 n136-128-154:823071:823265 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1 n136-128-154:823071:823265 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1 n136-128-154:823071:823265 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1 n136-128-154:823071:823265 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1 n136-128-154:823071:823265 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1 n136-128-154:823071:823265 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1 n136-128-154:823071:823265 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1 n136-128-154:823071:823265 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1 n136-128-154:823070:823256 [0] NCCL INFO Channel 00/24 : 0 1 n136-128-154:823070:823256 [0] NCCL INFO Channel 01/24 : 0 1 n136-128-154:823070:823256 [0] NCCL INFO Channel 02/24 : 0 1 n136-128-154:823071:823265 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0 n136-128-154:823070:823256 [0] NCCL INFO Channel 03/24 : 0 1 n136-128-154:823071:823265 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0 n136-128-154:823070:823256 [0] NCCL INFO Channel 04/24 : 0 1 n136-128-154:823071:823265 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0 n136-128-154:823070:823256 [0] NCCL INFO Channel 05/24 : 0 1 n136-128-154:823071:823265 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0 n136-128-154:823070:823256 [0] NCCL INFO Channel 06/24 : 0 1 n136-128-154:823071:823265 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0 n136-128-154:823070:823256 [0] NCCL INFO Channel 07/24 : 0 1 n136-128-154:823071:823265 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0 n136-128-154:823070:823256 [0] NCCL INFO Channel 08/24 : 0 1 n136-128-154:823071:823265 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0 n136-128-154:823070:823256 [0] NCCL INFO Channel 09/24 : 0 1 n136-128-154:823071:823265 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0 n136-128-154:823070:823256 [0] NCCL INFO Channel 10/24 : 0 1 n136-128-154:823071:823265 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0 n136-128-154:823070:823256 [0] NCCL INFO Channel 11/24 : 0 1 n136-128-154:823071:823265 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0 n136-128-154:823070:823256 [0] NCCL INFO Channel 12/24 : 0 1 n136-128-154:823071:823265 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0 n136-128-154:823070:823256 [0] NCCL INFO Channel 13/24 : 0 1 n136-128-154:823071:823265 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0 n136-128-154:823070:823256 [0] NCCL INFO Channel 14/24 : 0 1 n136-128-154:823071:823265 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0 n136-128-154:823070:823256 [0] NCCL INFO Channel 15/24 : 0 1 n136-128-154:823071:823265 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0 n136-128-154:823070:823256 [0] NCCL INFO Channel 16/24 : 0 1 n136-128-154:823071:823265 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0 n136-128-154:823070:823256 [0] NCCL INFO Channel 17/24 : 0 1 n136-128-154:823071:823265 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0 n136-128-154:823070:823256 [0] NCCL INFO Channel 18/24 : 0 1 n136-128-154:823071:823265 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0 n136-128-154:823070:823256 [0] NCCL INFO Channel 19/24 : 0 1 n136-128-154:823071:823265 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0 n136-128-154:823070:823256 [0] NCCL INFO Channel 20/24 : 0 1 n136-128-154:823071:823265 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0 n136-128-154:823070:823256 [0] NCCL INFO Channel 21/24 : 0 1 n136-128-154:823071:823265 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0 n136-128-154:823070:823256 [0] NCCL INFO Channel 22/24 : 0 1 n136-128-154:823071:823265 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0 n136-128-154:823070:823256 [0] NCCL INFO Channel 23/24 : 0 1 n136-128-154:823071:823265 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0 n136-128-154:823070:823256 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1 n136-128-154:823071:823265 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0 n136-128-154:823070:823256 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1 n136-128-154:823071:823265 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0 n136-128-154:823070:823256 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1 n136-128-154:823071:823265 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1 n136-128-154:823070:823256 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1 n136-128-154:823070:823256 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1 n136-128-154:823071:823265 [1] NCCL INFO P2P Chunksize set to 524288 n136-128-154:823070:823256 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1 n136-128-154:823070:823256 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1 n136-128-154:823070:823256 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1 n136-128-154:823070:823256 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1 n136-128-154:823070:823256 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1 n136-128-154:823070:823256 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1 n136-128-154:823070:823256 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1 n136-128-154:823070:823256 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1 n136-128-154:823070:823256 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1 n136-128-154:823070:823256 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1 n136-128-154:823070:823256 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1 n136-128-154:823070:823256 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1 n136-128-154:823070:823256 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1 n136-128-154:823070:823256 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1 n136-128-154:823070:823256 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1 n136-128-154:823070:823256 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1 n136-128-154:823070:823256 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1 n136-128-154:823070:823256 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1 n136-128-154:823070:823256 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1 n136-128-154:823070:823256 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1 n136-128-154:823070:823256 [0] NCCL INFO P2P Chunksize set to 524288 n136-128-154:823071:823265 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. n136-128-154:823071:823272 [1] NCCL INFO [Proxy Service] Device 1 CPU core 98 n136-128-154:823071:823273 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 99 n136-128-154:823070:823256 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. n136-128-154:823070:823256 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 n136-128-154:823070:823274 [0] NCCL INFO [Proxy Service] Device 0 CPU core 37 n136-128-154:823070:823275 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 38 n136-128-154:823071:823265 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 n136-128-154:823071:823265 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer n136-128-154:823070:823256 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 n136-128-154:823070:823256 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer n136-128-154:823070:823256 [0] NCCL INFO CC Off, workFifoBytes 1048576 n136-128-154:823071:823265 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. n136-128-154:823071:823265 [1] NCCL INFO ncclCommInitRankConfig comm 0x1199ce00 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0xc7f0e4b2f8d4055b - Init COMPLETE n136-128-154:823071:823265 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.53 (kernels 0.12, alloc 0.23, bootstrap 0.04, allgathers 0.00, topo 0.05, graphs 0.00, connections 0.01, rest 0.08) n136-128-154:823070:823256 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. n136-128-154:823070:823256 [0] NCCL INFO ncclCommInitRankConfig comm 0x11703740 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0xc7f0e4b2f8d4055b - Init COMPLETE n136-128-154:823070:823256 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 2.12 (kernels 0.14, alloc 0.20, bootstrap 1.64, allgathers 0.00, topo 0.05, graphs 0.00, connections 0.01, rest 0.09) n136-128-154:823071:823277 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823277 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823277 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823277 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823277 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823277 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823071:823277 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823071:823277 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823071:823277 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823071:823277 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823071:823277 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823071:823277 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823071:823277 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823071:823277 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823071:823277 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823071:823277 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823071:823277 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823071:823277 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823071:823277 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823071:823277 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823071:823277 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823071:823277 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823071:823277 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823071:823277 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823070:823278 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 n136-128-154:823071:823277 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 2025-12-01:23:17:26 INFO [evaluator:559] Running loglikelihood requests 2025-12-01:23:17:26 INFO [evaluator:559] Running loglikelihood requests Running loglikelihood requests: 0%| | 0/2066 [00:00 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823071:823454 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization' To add an exception for this directory, call: git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization n136-128-154:823071:823460 [1] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:823071:823460 [1] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:823071:823460 [1] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:823071:823460 [1] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:823071:823460 [1] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:823071:823460 [1] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:823071:823272 [1] NCCL INFO misc/socket.cc:915 -> 3 2025-12-01:23:20:01 INFO [loggers.evaluation_tracker:209] Saving results aggregated hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-32B), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto (64) | Tasks |Version|Filter|n-shot|Metric| |Value | |Stderr| |--------------|------:|------|-----:|------|---|-----:|---|-----:| |truthfulqa_mc1| 2|none | 0|acc |↑ |0.4015|± |0.0172| [rank0]:[W1201 23:20:02.397062022 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) n136-128-154:823070:823509 [0] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:823070:823509 [0] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:823070:823509 [0] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:823070:823274 [0] NCCL INFO misc/socket.cc:915 -> 3 n136-128-154:823070:823509 [0] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:823070:823509 [0] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:823070:823509 [0] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:823071:823272 [1] NCCL INFO misc/socket.cc:915 -> 3 n136-128-154:823071:823460 [1] NCCL INFO comm 0x1199ce00 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE n136-128-154:823070:823509 [0] NCCL INFO comm 0x11703740 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE 任务 truthfulqa_mc1 评估完成! ================================================== 开始评估:任务=hellaswag | 少样本数=10 | 模型=Qwen2.5-32B 输出路径:results2/Qwen2.5-32B/base_hellaswag.json ================================================== The following values were not passed to `accelerate launch` and had defaults used instead: More than one GPU was found, enabling multi-GPU training. If this was unintended please pass in `--num_processes=1`. `--num_machines` was set to a value of `1` `--mixed_precision` was set to a value of `'no'` `--dynamo_backend` was set to a value of `'no'` To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`. 2025-12-01:23:20:51 INFO [__main__:440] Selected Tasks: ['hellaswag'] 2025-12-01:23:20:51 INFO [__main__:440] Selected Tasks: ['hellaswag'] 2025-12-01:23:20:51 INFO [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2025-12-01:23:20:51 INFO [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-32B'} 2025-12-01:23:20:51 INFO [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2025-12-01:23:20:51 INFO [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-32B'} 2025-12-01:23:20:52 WARNING [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. 2025-12-01:23:20:52 INFO [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'} 2025-12-01:23:20:52 INFO [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'} `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! Loading checkpoint shards: 0%| | 0/17 [00:00 n136-128-154:823558:823786 [1] NCCL INFO Initialized NET plugin IB n136-128-154:823557:823785 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. n136-128-154:823557:823785 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0> n136-128-154:823557:823785 [0] NCCL INFO Initialized NET plugin IB n136-128-154:823558:823786 [1] NCCL INFO Assigned NET plugin IB to comm n136-128-154:823558:823786 [1] NCCL INFO Using network IB n136-128-154:823557:823785 [0] NCCL INFO Assigned NET plugin IB to comm n136-128-154:823557:823785 [0] NCCL INFO Using network IB n136-128-154:823558:823786 [1] NCCL INFO ncclCommInitRankConfig comm 0x122852c0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x496e757c0d64cca9 - Init START n136-128-154:823557:823785 [0] NCCL INFO ncclCommInitRankConfig comm 0x531ff3d0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x496e757c0d64cca9 - Init START n136-128-154:823557:823785 [0] NCCL INFO RAS client listening socket at ::1<28028> n136-128-154:823558:823786 [1] NCCL INFO RAS client listening socket at ::1<28028> n136-128-154:823557:823785 [0] NCCL INFO TOPO/NET : Importing network plugins to topology n136-128-154:823557:823785 [0] NCCL INFO Retrieving state for IB n136-128-154:823557:823785 [0] NCCL INFO Initialized state 0 for IB n136-128-154:823557:823785 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null) n136-128-154:823558:823786 [1] NCCL INFO TOPO/NET : Importing network plugins to topology n136-128-154:823558:823786 [1] NCCL INFO Retrieving state for IB n136-128-154:823558:823786 [1] NCCL INFO Initialized state 0 for IB n136-128-154:823557:823785 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null) n136-128-154:823558:823786 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null) n136-128-154:823557:823785 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null) n136-128-154:823558:823786 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null) n136-128-154:823557:823785 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null) n136-128-154:823558:823786 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null) n136-128-154:823558:823786 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null) n136-128-154:823558:823786 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default n136-128-154:823558:823786 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default n136-128-154:823557:823785 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 3 (distance 5 <= 5), read 0 mode Default n136-128-154:823557:823785 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 3 (distance 5 <= 5), read 0 mode Default n136-128-154:823558:823786 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 === n136-128-154:823558:823786 [1] NCCL INFO CPU/0-1 (1/1/2) n136-128-154:823558:823786 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff) n136-128-154:823558:823786 [1] NCCL INFO + PCI[24.0] - NIC/0-95000 n136-128-154:823558:823786 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff) n136-128-154:823558:823786 [1] NCCL INFO + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8) n136-128-154:823558:823786 [1] NCCL INFO + PCI[24.0] - GPU/0-c5000 (0) n136-128-154:823558:823786 [1] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:823558:823786 [1] NCCL INFO + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8) n136-128-154:823558:823786 [1] NCCL INFO + PCI[24.0] - GPU/0-c9000 (1) n136-128-154:823558:823786 [1] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:823558:823786 [1] NCCL INFO + PCI[24.0] - NIC/0-cc000 n136-128-154:823558:823786 [1] NCCL INFO + SYS[10.0] - CPU/0-0 n136-128-154:823558:823786 [1] NCCL INFO CPU/0-0 (1/1/2) n136-128-154:823558:823786 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff) n136-128-154:823558:823786 [1] NCCL INFO + PCI[24.0] - NIC/0-1d000 n136-128-154:823558:823786 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff) n136-128-154:823558:823786 [1] NCCL INFO + PCI[24.0] - NIC/0-60000 n136-128-154:823558:823786 [1] NCCL INFO + SYS[10.0] - CPU/0-1 n136-128-154:823558:823786 [1] NCCL INFO ========================================== n136-128-154:823558:823786 [1] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:823558:823786 [1] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:823558:823786 [1] NCCL INFO Setting affinity for GPU 7 to 33-62,97-126 n136-128-154:823557:823785 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 === n136-128-154:823557:823785 [0] NCCL INFO CPU/0-1 (1/1/2) n136-128-154:823557:823785 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff) n136-128-154:823557:823785 [0] NCCL INFO + PCI[24.0] - NIC/0-95000 n136-128-154:823557:823785 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff) n136-128-154:823557:823785 [0] NCCL INFO + PCI[24.0] - PCI/0-c3000 (1000c01010de13b8) n136-128-154:823557:823785 [0] NCCL INFO + PCI[24.0] - GPU/0-c5000 (0) n136-128-154:823557:823785 [0] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:823557:823785 [0] NCCL INFO + PCI[24.0] - PCI/0-c7000 (1000c01010de13b8) n136-128-154:823557:823785 [0] NCCL INFO + PCI[24.0] - GPU/0-c9000 (1) n136-128-154:823557:823785 [0] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:823557:823785 [0] NCCL INFO + PCI[24.0] - NIC/0-cc000 n136-128-154:823557:823785 [0] NCCL INFO + SYS[10.0] - CPU/0-0 n136-128-154:823557:823785 [0] NCCL INFO CPU/0-0 (1/1/2) n136-128-154:823557:823785 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff) n136-128-154:823558:823786 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1 n136-128-154:823557:823785 [0] NCCL INFO + PCI[24.0] - NIC/0-1d000 n136-128-154:823558:823786 [1] NCCL INFO 0 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff) n136-128-154:823558:823786 [1] NCCL INFO 1 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO + PCI[24.0] - NIC/0-60000 n136-128-154:823558:823786 [1] NCCL INFO 2 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO + SYS[10.0] - CPU/0-1 n136-128-154:823558:823786 [1] NCCL INFO 3 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO ========================================== n136-128-154:823558:823786 [1] NCCL INFO 4 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823558:823786 [1] NCCL INFO 5 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823558:823786 [1] NCCL INFO 6 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO GPU/0-c5000 :GPU/0-c5000 (0/5000.0/LOC) GPU/0-c9000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:823558:823786 [1] NCCL INFO 7 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823558:823786 [1] NCCL INFO 8 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO GPU/0-c9000 :GPU/0-c5000 (2/240.0/NVL) GPU/0-c9000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:823558:823786 [1] NCCL INFO 9 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823558:823786 [1] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823558:823786 [1] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO Setting affinity for GPU 6 to 33-62,97-126 n136-128-154:823558:823786 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0 n136-128-154:823558:823786 [1] NCCL INFO 0 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823558:823786 [1] NCCL INFO 1 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823558:823786 [1] NCCL INFO 2 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823558:823786 [1] NCCL INFO 3 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823558:823786 [1] NCCL INFO 4 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823558:823786 [1] NCCL INFO 5 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823558:823786 [1] NCCL INFO 6 : GPU/0-c9000 GPU/0-c5000 n136-128-154:823558:823786 [1] NCCL INFO 7 : GPU/0-c9000 GPU/0-c5000 n136-128-154:823558:823786 [1] NCCL INFO 8 : GPU/0-c9000 GPU/0-c5000 n136-128-154:823558:823786 [1] NCCL INFO 9 : GPU/0-c9000 GPU/0-c5000 n136-128-154:823558:823786 [1] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000 n136-128-154:823558:823786 [1] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000 n136-128-154:823557:823785 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1 n136-128-154:823557:823785 [0] NCCL INFO 0 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO 1 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO 2 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO 3 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO 4 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO 5 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO 6 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO 7 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO 8 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO 9 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO 10 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO 11 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0 n136-128-154:823557:823785 [0] NCCL INFO 0 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO 1 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO 2 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO 3 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO 4 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO 5 : GPU/0-c5000 GPU/0-c9000 n136-128-154:823557:823785 [0] NCCL INFO 6 : GPU/0-c9000 GPU/0-c5000 n136-128-154:823557:823785 [0] NCCL INFO 7 : GPU/0-c9000 GPU/0-c5000 n136-128-154:823557:823785 [0] NCCL INFO 8 : GPU/0-c9000 GPU/0-c5000 n136-128-154:823557:823785 [0] NCCL INFO 9 : GPU/0-c9000 GPU/0-c5000 n136-128-154:823557:823785 [0] NCCL INFO 10 : GPU/0-c9000 GPU/0-c5000 n136-128-154:823557:823785 [0] NCCL INFO 11 : GPU/0-c9000 GPU/0-c5000 n136-128-154:823558:823786 [1] NCCL INFO comm 0x122852c0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0 n136-128-154:823557:823785 [0] NCCL INFO comm 0x531ff3d0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0 n136-128-154:823558:823786 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1 n136-128-154:823558:823786 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1 n136-128-154:823558:823786 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1 n136-128-154:823558:823786 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1 n136-128-154:823558:823786 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1 n136-128-154:823558:823786 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1 n136-128-154:823558:823786 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1 n136-128-154:823558:823786 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1 n136-128-154:823558:823786 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1 n136-128-154:823558:823786 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1 n136-128-154:823558:823786 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1 n136-128-154:823558:823786 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1 n136-128-154:823558:823786 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1 n136-128-154:823558:823786 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1 n136-128-154:823558:823786 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1 n136-128-154:823558:823786 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1 n136-128-154:823558:823786 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1 n136-128-154:823558:823786 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1 n136-128-154:823558:823786 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1 n136-128-154:823558:823786 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1 n136-128-154:823558:823786 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1 n136-128-154:823558:823786 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1 n136-128-154:823558:823786 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1 n136-128-154:823558:823786 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1 n136-128-154:823557:823785 [0] NCCL INFO Channel 00/24 : 0 1 n136-128-154:823557:823785 [0] NCCL INFO Channel 01/24 : 0 1 n136-128-154:823558:823786 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0 n136-128-154:823557:823785 [0] NCCL INFO Channel 02/24 : 0 1 n136-128-154:823558:823786 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0 n136-128-154:823557:823785 [0] NCCL INFO Channel 03/24 : 0 1 n136-128-154:823558:823786 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0 n136-128-154:823557:823785 [0] NCCL INFO Channel 04/24 : 0 1 n136-128-154:823558:823786 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0 n136-128-154:823557:823785 [0] NCCL INFO Channel 05/24 : 0 1 n136-128-154:823558:823786 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0 n136-128-154:823557:823785 [0] NCCL INFO Channel 06/24 : 0 1 n136-128-154:823558:823786 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0 n136-128-154:823557:823785 [0] NCCL INFO Channel 07/24 : 0 1 n136-128-154:823558:823786 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0 n136-128-154:823557:823785 [0] NCCL INFO Channel 08/24 : 0 1 n136-128-154:823558:823786 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0 n136-128-154:823557:823785 [0] NCCL INFO Channel 09/24 : 0 1 n136-128-154:823558:823786 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0 n136-128-154:823557:823785 [0] NCCL INFO Channel 10/24 : 0 1 n136-128-154:823558:823786 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0 n136-128-154:823557:823785 [0] NCCL INFO Channel 11/24 : 0 1 n136-128-154:823558:823786 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0 n136-128-154:823557:823785 [0] NCCL INFO Channel 12/24 : 0 1 n136-128-154:823558:823786 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0 n136-128-154:823557:823785 [0] NCCL INFO Channel 13/24 : 0 1 n136-128-154:823558:823786 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0 n136-128-154:823557:823785 [0] NCCL INFO Channel 14/24 : 0 1 n136-128-154:823558:823786 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0 n136-128-154:823557:823785 [0] NCCL INFO Channel 15/24 : 0 1 n136-128-154:823558:823786 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0 n136-128-154:823557:823785 [0] NCCL INFO Channel 16/24 : 0 1 n136-128-154:823558:823786 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0 n136-128-154:823557:823785 [0] NCCL INFO Channel 17/24 : 0 1 n136-128-154:823558:823786 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0 n136-128-154:823557:823785 [0] NCCL INFO Channel 18/24 : 0 1 n136-128-154:823558:823786 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0 n136-128-154:823557:823785 [0] NCCL INFO Channel 19/24 : 0 1 n136-128-154:823558:823786 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0 n136-128-154:823557:823785 [0] NCCL INFO Channel 20/24 : 0 1 n136-128-154:823558:823786 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0 n136-128-154:823557:823785 [0] NCCL INFO Channel 21/24 : 0 1 n136-128-154:823558:823786 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0 n136-128-154:823557:823785 [0] NCCL INFO Channel 22/24 : 0 1 n136-128-154:823558:823786 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0 n136-128-154:823557:823785 [0] NCCL INFO Channel 23/24 : 0 1 n136-128-154:823558:823786 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0 n136-128-154:823557:823785 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1 n136-128-154:823558:823786 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0 n136-128-154:823557:823785 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1 n136-128-154:823558:823786 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1 n136-128-154:823557:823785 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1 n136-128-154:823557:823785 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1 n136-128-154:823558:823786 [1] NCCL INFO P2P Chunksize set to 524288 n136-128-154:823557:823785 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1 n136-128-154:823557:823785 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1 n136-128-154:823557:823785 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1 n136-128-154:823557:823785 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1 n136-128-154:823557:823785 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1 n136-128-154:823557:823785 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1 n136-128-154:823557:823785 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1 n136-128-154:823557:823785 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1 n136-128-154:823557:823785 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1 n136-128-154:823557:823785 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1 n136-128-154:823557:823785 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1 n136-128-154:823557:823785 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1 n136-128-154:823557:823785 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1 n136-128-154:823557:823785 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1 n136-128-154:823557:823785 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1 n136-128-154:823557:823785 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1 n136-128-154:823557:823785 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1 n136-128-154:823557:823785 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1 n136-128-154:823557:823785 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1 n136-128-154:823557:823785 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1 n136-128-154:823557:823785 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1 n136-128-154:823557:823785 [0] NCCL INFO P2P Chunksize set to 524288 n136-128-154:823558:823786 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. n136-128-154:823558:823797 [1] NCCL INFO [Proxy Service] Device 1 CPU core 99 n136-128-154:823558:823798 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 39 n136-128-154:823557:823785 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. n136-128-154:823557:823785 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 n136-128-154:823557:823799 [0] NCCL INFO [Proxy Service] Device 0 CPU core 105 n136-128-154:823557:823800 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 106 n136-128-154:823558:823786 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 n136-128-154:823558:823786 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer n136-128-154:823557:823785 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 n136-128-154:823557:823785 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer n136-128-154:823557:823785 [0] NCCL INFO CC Off, workFifoBytes 1048576 n136-128-154:823558:823786 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. n136-128-154:823558:823786 [1] NCCL INFO ncclCommInitRankConfig comm 0x122852c0 rank 1 nranks 2 cudaDev 1 nvmlDev 7 busId c9000 commId 0x496e757c0d64cca9 - Init COMPLETE n136-128-154:823558:823786 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.45 (kernels 0.16, alloc 0.20, bootstrap 0.00, allgathers 0.00, topo 0.05, graphs 0.00, connections 0.01, rest 0.02) n136-128-154:823557:823785 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. n136-128-154:823557:823785 [0] NCCL INFO ncclCommInitRankConfig comm 0x531ff3d0 rank 0 nranks 2 cudaDev 0 nvmlDev 6 busId c5000 commId 0x496e757c0d64cca9 - Init COMPLETE n136-128-154:823557:823785 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.46 (kernels 0.17, alloc 0.20, bootstrap 0.00, allgathers 0.00, topo 0.05, graphs 0.00, connections 0.01, rest 0.02) n136-128-154:823558:823801 [1] NCCL INFO Channel 00/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:823801 [1] NCCL INFO Channel 01/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:823801 [1] NCCL INFO Channel 02/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:823801 [1] NCCL INFO Channel 03/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:823801 [1] NCCL INFO Channel 04/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:823801 [1] NCCL INFO Channel 05/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:823801 [1] NCCL INFO Channel 06/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823557:823802 [0] NCCL INFO Channel 00/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823558:823801 [1] NCCL INFO Channel 07/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823557:823802 [0] NCCL INFO Channel 01/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823558:823801 [1] NCCL INFO Channel 08/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823557:823802 [0] NCCL INFO Channel 02/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823558:823801 [1] NCCL INFO Channel 09/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823557:823802 [0] NCCL INFO Channel 03/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823558:823801 [1] NCCL INFO Channel 10/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823557:823802 [0] NCCL INFO Channel 04/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823558:823801 [1] NCCL INFO Channel 11/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823557:823802 [0] NCCL INFO Channel 05/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823558:823801 [1] NCCL INFO Channel 12/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823557:823802 [0] NCCL INFO Channel 06/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823558:823801 [1] NCCL INFO Channel 13/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823557:823802 [0] NCCL INFO Channel 07/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823558:823801 [1] NCCL INFO Channel 14/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823557:823802 [0] NCCL INFO Channel 08/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823558:823801 [1] NCCL INFO Channel 15/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823557:823802 [0] NCCL INFO Channel 09/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823558:823801 [1] NCCL INFO Channel 16/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823557:823802 [0] NCCL INFO Channel 10/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823558:823801 [1] NCCL INFO Channel 17/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823557:823802 [0] NCCL INFO Channel 11/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823558:823801 [1] NCCL INFO Channel 18/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823557:823802 [0] NCCL INFO Channel 12/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823558:823801 [1] NCCL INFO Channel 19/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823557:823802 [0] NCCL INFO Channel 13/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823558:823801 [1] NCCL INFO Channel 20/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823557:823802 [0] NCCL INFO Channel 14/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823558:823801 [1] NCCL INFO Channel 21/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823557:823802 [0] NCCL INFO Channel 15/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823558:823801 [1] NCCL INFO Channel 22/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823557:823802 [0] NCCL INFO Channel 16/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823558:823801 [1] NCCL INFO Channel 23/0 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823557:823802 [0] NCCL INFO Channel 17/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823557:823802 [0] NCCL INFO Channel 18/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823557:823802 [0] NCCL INFO Channel 19/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823557:823802 [0] NCCL INFO Channel 20/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823557:823802 [0] NCCL INFO Channel 21/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823557:823802 [0] NCCL INFO Channel 22/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823557:823802 [0] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/CUMEM/read n136-128-154:823558:823801 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 n136-128-154:823557:823802 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 2025-12-01:23:22:01 INFO [evaluator:559] Running loglikelihood requests 2025-12-01:23:22:01 INFO [evaluator:559] Running loglikelihood requests Running loglikelihood requests: 0%| | 0/20084 [00:00 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 01/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 02/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 03/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 04/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 05/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 06/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 07/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 08/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 09/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 10/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 11/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 12/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 13/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 14/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 15/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 16/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 17/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 18/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 19/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 20/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 21/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 22/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 23/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 24/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 25/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 26/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 27/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 28/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 29/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 30/1 : 1[7] -> 0[6] via P2P/CUMEM/read n136-128-154:823558:829820 [1] NCCL INFO Channel 31/1 : 1[7] -> 0[6] via P2P/CUMEM/read fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization' To add an exception for this directory, call: git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization n136-128-154:823558:829826 [1] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:823558:829826 [1] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:823558:829826 [1] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:823558:829826 [1] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:823558:829826 [1] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:823558:829826 [1] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:823558:823797 [1] NCCL INFO misc/socket.cc:915 -> 3 2025-12-02:00:52:01 INFO [loggers.evaluation_tracker:209] Saving results aggregated hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-32B), gen_kwargs: (None), limit: None, num_fewshot: 10, batch_size: auto (11) | Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr| |---------|------:|------|-----:|--------|---|-----:|---|-----:| |hellaswag| 1|none | 10|acc |↑ |0.6586|± |0.0047| | | |none | 10|acc_norm|↑ |0.8505|± |0.0036| [rank0]:[W1202 00:52:02.618550941 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) n136-128-154:823557:829876 [0] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:823557:829876 [0] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:823557:829876 [0] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:823557:823799 [0] NCCL INFO misc/socket.cc:915 -> 3 n136-128-154:823557:829876 [0] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:823558:823797 [1] NCCL INFO misc/socket.cc:915 -> 3 n136-128-154:823557:829876 [0] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:823557:829876 [0] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:823558:829826 [1] NCCL INFO comm 0x122852c0 rank 1 nranks 2 cudaDev 1 busId c9000 - Abort COMPLETE n136-128-154:823557:829876 [0] NCCL INFO comm 0x531ff3d0 rank 0 nranks 2 cudaDev 0 busId c5000 - Abort COMPLETE 任务 hellaswag 评估完成!