nohup: ignoring input ================================================== 开始评估:任务=triviaqa | 少样本数=0 | 模型=Qwen2.5-14B 输出路径:results2/Qwen2.5-14B/base_triviaqa.json ================================================== The following values were not passed to `accelerate launch` and had defaults used instead: More than one GPU was found, enabling multi-GPU training. If this was unintended please pass in `--num_processes=1`. `--num_machines` was set to a value of `1` `--mixed_precision` was set to a value of `'no'` `--dynamo_backend` was set to a value of `'no'` To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`. 2025-12-01:19:45:57 INFO [__main__:440] Selected Tasks: ['triviaqa'] 2025-12-01:19:45:57 INFO [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2025-12-01:19:45:57 INFO [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-14B'} 2025-12-01:19:45:58 INFO [__main__:440] Selected Tasks: ['triviaqa'] 2025-12-01:19:45:58 INFO [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2025-12-01:19:45:58 INFO [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-14B'} 2025-12-01:19:46:01 WARNING [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. 2025-12-01:19:46:01 INFO [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'} `torch_dtype` is deprecated! Use `dtype` instead! 2025-12-01:19:46:01 INFO [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'} `torch_dtype` is deprecated! Use `dtype` instead! Loading checkpoint shards: 0%| | 0/8 [00:00 n136-128-154:773514:774459 [0] NCCL INFO Initialized NET plugin IB n136-128-154:773515:774460 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. n136-128-154:773515:774460 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0> n136-128-154:773515:774460 [1] NCCL INFO Initialized NET plugin IB n136-128-154:773514:774459 [0] NCCL INFO Assigned NET plugin IB to comm n136-128-154:773514:774459 [0] NCCL INFO Using network IB n136-128-154:773515:774460 [1] NCCL INFO Assigned NET plugin IB to comm n136-128-154:773515:774460 [1] NCCL INFO Using network IB n136-128-154:773515:774460 [1] NCCL INFO ncclCommInitRankConfig comm 0x11550b80 rank 1 nranks 2 cudaDev 1 nvmlDev 5 busId 8e000 commId 0xb7d48819bedd2d1b - Init START n136-128-154:773514:774459 [0] NCCL INFO ncclCommInitRankConfig comm 0x1135d5c0 rank 0 nranks 2 cudaDev 0 nvmlDev 4 busId 89000 commId 0xb7d48819bedd2d1b - Init START n136-128-154:773515:774460 [1] NCCL INFO RAS client listening socket at ::1<28028> n136-128-154:773514:774459 [0] NCCL INFO RAS client listening socket at ::1<28028> n136-128-154:773515:774460 [1] NCCL INFO TOPO/NET : Importing network plugins to topology n136-128-154:773515:774460 [1] NCCL INFO Retrieving state for IB n136-128-154:773515:774460 [1] NCCL INFO Initialized state 0 for IB n136-128-154:773515:774460 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null) n136-128-154:773515:774460 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null) n136-128-154:773515:774460 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null) n136-128-154:773515:774460 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null) n136-128-154:773514:774459 [0] NCCL INFO TOPO/NET : Importing network plugins to topology n136-128-154:773514:774459 [0] NCCL INFO Retrieving state for IB n136-128-154:773514:774459 [0] NCCL INFO Initialized state 0 for IB n136-128-154:773514:774459 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null) n136-128-154:773514:774459 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null) n136-128-154:773514:774459 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null) n136-128-154:773514:774459 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null) n136-128-154:773515:774460 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 2 (distance 5 <= 5), read 0 mode Default n136-128-154:773515:774460 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 2 (distance 5 <= 5), read 0 mode Default n136-128-154:773514:774459 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 2 (distance 5 <= 5), read 0 mode Default n136-128-154:773514:774459 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 2 (distance 5 <= 5), read 0 mode Default n136-128-154:773515:774460 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 === n136-128-154:773515:774460 [1] NCCL INFO CPU/0-1 (1/1/2) n136-128-154:773514:774459 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 === n136-128-154:773515:774460 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff) n136-128-154:773514:774459 [0] NCCL INFO CPU/0-1 (1/1/2) n136-128-154:773515:774460 [1] NCCL INFO + PCI[24.0] - PCI/0-87000 (1000c01010de13b8) n136-128-154:773514:774459 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff) n136-128-154:773515:774460 [1] NCCL INFO + PCI[24.0] - GPU/0-89000 (0) n136-128-154:773514:774459 [0] NCCL INFO + PCI[24.0] - PCI/0-87000 (1000c01010de13b8) n136-128-154:773515:774460 [1] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:773514:774459 [0] NCCL INFO + PCI[24.0] - GPU/0-89000 (0) n136-128-154:773515:774460 [1] NCCL INFO + PCI[24.0] - NIC/0-95000 n136-128-154:773514:774459 [0] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:773515:774460 [1] NCCL INFO + PCI[24.0] - PCI/0-8c000 (1000c01010de13b8) n136-128-154:773514:774459 [0] NCCL INFO + PCI[24.0] - NIC/0-95000 n136-128-154:773515:774460 [1] NCCL INFO + PCI[24.0] - GPU/0-8e000 (1) n136-128-154:773514:774459 [0] NCCL INFO + PCI[24.0] - PCI/0-8c000 (1000c01010de13b8) n136-128-154:773515:774460 [1] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:773514:774459 [0] NCCL INFO + PCI[24.0] - GPU/0-8e000 (1) n136-128-154:773515:774460 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff) n136-128-154:773514:774459 [0] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:773515:774460 [1] NCCL INFO + PCI[24.0] - NIC/0-cc000 n136-128-154:773514:774459 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff) n136-128-154:773515:774460 [1] NCCL INFO + SYS[10.0] - CPU/0-0 n136-128-154:773514:774459 [0] NCCL INFO + PCI[24.0] - NIC/0-cc000 n136-128-154:773515:774460 [1] NCCL INFO CPU/0-0 (1/1/2) n136-128-154:773514:774459 [0] NCCL INFO + SYS[10.0] - CPU/0-0 n136-128-154:773515:774460 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff) n136-128-154:773514:774459 [0] NCCL INFO CPU/0-0 (1/1/2) n136-128-154:773515:774460 [1] NCCL INFO + PCI[24.0] - NIC/0-1d000 n136-128-154:773514:774459 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff) n136-128-154:773515:774460 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff) n136-128-154:773514:774459 [0] NCCL INFO + PCI[24.0] - NIC/0-1d000 n136-128-154:773515:774460 [1] NCCL INFO + PCI[24.0] - NIC/0-60000 n136-128-154:773514:774459 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff) n136-128-154:773515:774460 [1] NCCL INFO + SYS[10.0] - CPU/0-1 n136-128-154:773514:774459 [0] NCCL INFO + PCI[24.0] - NIC/0-60000 n136-128-154:773515:774460 [1] NCCL INFO ========================================== n136-128-154:773514:774459 [0] NCCL INFO + SYS[10.0] - CPU/0-1 n136-128-154:773514:774459 [0] NCCL INFO ========================================== n136-128-154:773515:774460 [1] NCCL INFO GPU/0-89000 :GPU/0-89000 (0/5000.0/LOC) GPU/0-8e000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:773514:774459 [0] NCCL INFO GPU/0-89000 :GPU/0-89000 (0/5000.0/LOC) GPU/0-8e000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:773515:774460 [1] NCCL INFO GPU/0-8e000 :GPU/0-89000 (2/240.0/NVL) GPU/0-8e000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:773514:774459 [0] NCCL INFO GPU/0-8e000 :GPU/0-89000 (2/240.0/NVL) GPU/0-8e000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:773515:774460 [1] NCCL INFO Setting affinity for GPU 5 to 33-62,97-126 n136-128-154:773514:774459 [0] NCCL INFO Setting affinity for GPU 4 to 33-62,97-126 n136-128-154:773515:774460 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1 n136-128-154:773515:774460 [1] NCCL INFO 0 : GPU/0-89000 GPU/0-8e000 n136-128-154:773515:774460 [1] NCCL INFO 1 : GPU/0-89000 GPU/0-8e000 n136-128-154:773515:774460 [1] NCCL INFO 2 : GPU/0-89000 GPU/0-8e000 n136-128-154:773515:774460 [1] NCCL INFO 3 : GPU/0-89000 GPU/0-8e000 n136-128-154:773515:774460 [1] NCCL INFO 4 : GPU/0-89000 GPU/0-8e000 n136-128-154:773515:774460 [1] NCCL INFO 5 : GPU/0-89000 GPU/0-8e000 n136-128-154:773515:774460 [1] NCCL INFO 6 : GPU/0-89000 GPU/0-8e000 n136-128-154:773515:774460 [1] NCCL INFO 7 : GPU/0-89000 GPU/0-8e000 n136-128-154:773515:774460 [1] NCCL INFO 8 : GPU/0-89000 GPU/0-8e000 n136-128-154:773515:774460 [1] NCCL INFO 9 : GPU/0-89000 GPU/0-8e000 n136-128-154:773515:774460 [1] NCCL INFO 10 : GPU/0-89000 GPU/0-8e000 n136-128-154:773515:774460 [1] NCCL INFO 11 : GPU/0-89000 GPU/0-8e000 n136-128-154:773515:774460 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0 n136-128-154:773515:774460 [1] NCCL INFO 0 : GPU/0-89000 GPU/0-8e000 n136-128-154:773514:774459 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1 n136-128-154:773515:774460 [1] NCCL INFO 1 : GPU/0-89000 GPU/0-8e000 n136-128-154:773515:774460 [1] NCCL INFO 2 : GPU/0-89000 GPU/0-8e000 n136-128-154:773514:774459 [0] NCCL INFO 0 : GPU/0-89000 GPU/0-8e000 n136-128-154:773515:774460 [1] NCCL INFO 3 : GPU/0-89000 GPU/0-8e000 n136-128-154:773514:774459 [0] NCCL INFO 1 : GPU/0-89000 GPU/0-8e000 n136-128-154:773515:774460 [1] NCCL INFO 4 : GPU/0-89000 GPU/0-8e000 n136-128-154:773514:774459 [0] NCCL INFO 2 : GPU/0-89000 GPU/0-8e000 n136-128-154:773515:774460 [1] NCCL INFO 5 : GPU/0-89000 GPU/0-8e000 n136-128-154:773514:774459 [0] NCCL INFO 3 : GPU/0-89000 GPU/0-8e000 n136-128-154:773515:774460 [1] NCCL INFO 6 : GPU/0-8e000 GPU/0-89000 n136-128-154:773514:774459 [0] NCCL INFO 4 : GPU/0-89000 GPU/0-8e000 n136-128-154:773515:774460 [1] NCCL INFO 7 : GPU/0-8e000 GPU/0-89000 n136-128-154:773514:774459 [0] NCCL INFO 5 : GPU/0-89000 GPU/0-8e000 n136-128-154:773515:774460 [1] NCCL INFO 8 : GPU/0-8e000 GPU/0-89000 n136-128-154:773514:774459 [0] NCCL INFO 6 : GPU/0-89000 GPU/0-8e000 n136-128-154:773515:774460 [1] NCCL INFO 9 : GPU/0-8e000 GPU/0-89000 n136-128-154:773514:774459 [0] NCCL INFO 7 : GPU/0-89000 GPU/0-8e000 n136-128-154:773515:774460 [1] NCCL INFO 10 : GPU/0-8e000 GPU/0-89000 n136-128-154:773514:774459 [0] NCCL INFO 8 : GPU/0-89000 GPU/0-8e000 n136-128-154:773515:774460 [1] NCCL INFO 11 : GPU/0-8e000 GPU/0-89000 n136-128-154:773514:774459 [0] NCCL INFO 9 : GPU/0-89000 GPU/0-8e000 n136-128-154:773514:774459 [0] NCCL INFO 10 : GPU/0-89000 GPU/0-8e000 n136-128-154:773514:774459 [0] NCCL INFO 11 : GPU/0-89000 GPU/0-8e000 n136-128-154:773514:774459 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0 n136-128-154:773514:774459 [0] NCCL INFO 0 : GPU/0-89000 GPU/0-8e000 n136-128-154:773514:774459 [0] NCCL INFO 1 : GPU/0-89000 GPU/0-8e000 n136-128-154:773514:774459 [0] NCCL INFO 2 : GPU/0-89000 GPU/0-8e000 n136-128-154:773514:774459 [0] NCCL INFO 3 : GPU/0-89000 GPU/0-8e000 n136-128-154:773514:774459 [0] NCCL INFO 4 : GPU/0-89000 GPU/0-8e000 n136-128-154:773514:774459 [0] NCCL INFO 5 : GPU/0-89000 GPU/0-8e000 n136-128-154:773514:774459 [0] NCCL INFO 6 : GPU/0-8e000 GPU/0-89000 n136-128-154:773514:774459 [0] NCCL INFO 7 : GPU/0-8e000 GPU/0-89000 n136-128-154:773514:774459 [0] NCCL INFO 8 : GPU/0-8e000 GPU/0-89000 n136-128-154:773514:774459 [0] NCCL INFO 9 : GPU/0-8e000 GPU/0-89000 n136-128-154:773514:774459 [0] NCCL INFO 10 : GPU/0-8e000 GPU/0-89000 n136-128-154:773514:774459 [0] NCCL INFO 11 : GPU/0-8e000 GPU/0-89000 n136-128-154:773515:774460 [1] NCCL INFO comm 0x11550b80 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0 n136-128-154:773514:774459 [0] NCCL INFO comm 0x1135d5c0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0 n136-128-154:773515:774460 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1 n136-128-154:773515:774460 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1 n136-128-154:773515:774460 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1 n136-128-154:773515:774460 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1 n136-128-154:773515:774460 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1 n136-128-154:773515:774460 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1 n136-128-154:773515:774460 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1 n136-128-154:773515:774460 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1 n136-128-154:773515:774460 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1 n136-128-154:773515:774460 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1 n136-128-154:773515:774460 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1 n136-128-154:773515:774460 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1 n136-128-154:773515:774460 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1 n136-128-154:773515:774460 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1 n136-128-154:773515:774460 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1 n136-128-154:773515:774460 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1 n136-128-154:773515:774460 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1 n136-128-154:773515:774460 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1 n136-128-154:773515:774460 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1 n136-128-154:773515:774460 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1 n136-128-154:773515:774460 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1 n136-128-154:773515:774460 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1 n136-128-154:773515:774460 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1 n136-128-154:773515:774460 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1 n136-128-154:773514:774459 [0] NCCL INFO Channel 00/24 : 0 1 n136-128-154:773515:774460 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0 n136-128-154:773514:774459 [0] NCCL INFO Channel 01/24 : 0 1 n136-128-154:773515:774460 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0 n136-128-154:773514:774459 [0] NCCL INFO Channel 02/24 : 0 1 n136-128-154:773515:774460 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0 n136-128-154:773514:774459 [0] NCCL INFO Channel 03/24 : 0 1 n136-128-154:773515:774460 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0 n136-128-154:773514:774459 [0] NCCL INFO Channel 04/24 : 0 1 n136-128-154:773515:774460 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0 n136-128-154:773514:774459 [0] NCCL INFO Channel 05/24 : 0 1 n136-128-154:773515:774460 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0 n136-128-154:773514:774459 [0] NCCL INFO Channel 06/24 : 0 1 n136-128-154:773515:774460 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0 n136-128-154:773514:774459 [0] NCCL INFO Channel 07/24 : 0 1 n136-128-154:773515:774460 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0 n136-128-154:773514:774459 [0] NCCL INFO Channel 08/24 : 0 1 n136-128-154:773515:774460 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0 n136-128-154:773514:774459 [0] NCCL INFO Channel 09/24 : 0 1 n136-128-154:773515:774460 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0 n136-128-154:773514:774459 [0] NCCL INFO Channel 10/24 : 0 1 n136-128-154:773515:774460 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0 n136-128-154:773514:774459 [0] NCCL INFO Channel 11/24 : 0 1 n136-128-154:773515:774460 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0 n136-128-154:773514:774459 [0] NCCL INFO Channel 12/24 : 0 1 n136-128-154:773515:774460 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0 n136-128-154:773514:774459 [0] NCCL INFO Channel 13/24 : 0 1 n136-128-154:773515:774460 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0 n136-128-154:773514:774459 [0] NCCL INFO Channel 14/24 : 0 1 n136-128-154:773515:774460 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0 n136-128-154:773514:774459 [0] NCCL INFO Channel 15/24 : 0 1 n136-128-154:773515:774460 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0 n136-128-154:773514:774459 [0] NCCL INFO Channel 16/24 : 0 1 n136-128-154:773515:774460 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0 n136-128-154:773514:774459 [0] NCCL INFO Channel 17/24 : 0 1 n136-128-154:773515:774460 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0 n136-128-154:773514:774459 [0] NCCL INFO Channel 18/24 : 0 1 n136-128-154:773515:774460 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0 n136-128-154:773514:774459 [0] NCCL INFO Channel 19/24 : 0 1 n136-128-154:773515:774460 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0 n136-128-154:773514:774459 [0] NCCL INFO Channel 20/24 : 0 1 n136-128-154:773515:774460 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0 n136-128-154:773514:774459 [0] NCCL INFO Channel 21/24 : 0 1 n136-128-154:773515:774460 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0 n136-128-154:773514:774459 [0] NCCL INFO Channel 22/24 : 0 1 n136-128-154:773515:774460 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0 n136-128-154:773514:774459 [0] NCCL INFO Channel 23/24 : 0 1 n136-128-154:773515:774460 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0 n136-128-154:773515:774460 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1 n136-128-154:773514:774459 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1 n136-128-154:773514:774459 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1 n136-128-154:773515:774460 [1] NCCL INFO P2P Chunksize set to 524288 n136-128-154:773514:774459 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1 n136-128-154:773514:774459 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1 n136-128-154:773514:774459 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1 n136-128-154:773514:774459 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1 n136-128-154:773514:774459 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1 n136-128-154:773514:774459 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1 n136-128-154:773514:774459 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1 n136-128-154:773514:774459 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1 n136-128-154:773514:774459 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1 n136-128-154:773514:774459 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1 n136-128-154:773514:774459 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1 n136-128-154:773514:774459 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1 n136-128-154:773514:774459 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1 n136-128-154:773514:774459 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1 n136-128-154:773514:774459 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1 n136-128-154:773514:774459 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1 n136-128-154:773514:774459 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1 n136-128-154:773514:774459 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1 n136-128-154:773514:774459 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1 n136-128-154:773514:774459 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1 n136-128-154:773514:774459 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1 n136-128-154:773514:774459 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1 n136-128-154:773514:774459 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1 n136-128-154:773514:774459 [0] NCCL INFO P2P Chunksize set to 524288 n136-128-154:773515:774460 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. n136-128-154:773515:774479 [1] NCCL INFO [Proxy Service] Device 1 CPU core 103 n136-128-154:773515:774480 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 40 n136-128-154:773514:774459 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. n136-128-154:773514:774459 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 n136-128-154:773514:774481 [0] NCCL INFO [Proxy Service] Device 0 CPU core 105 n136-128-154:773514:774482 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 106 n136-128-154:773515:774460 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 n136-128-154:773515:774460 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer n136-128-154:773514:774459 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 n136-128-154:773514:774459 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer n136-128-154:773514:774459 [0] NCCL INFO CC Off, workFifoBytes 1048576 n136-128-154:773514:774459 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. n136-128-154:773514:774459 [0] NCCL INFO ncclCommInitRankConfig comm 0x1135d5c0 rank 0 nranks 2 cudaDev 0 nvmlDev 4 busId 89000 commId 0xb7d48819bedd2d1b - Init COMPLETE n136-128-154:773514:774459 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.70 (kernels 0.12, alloc 0.39, bootstrap 0.00, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.03, rest 0.10) n136-128-154:773515:774460 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. n136-128-154:773515:774460 [1] NCCL INFO ncclCommInitRankConfig comm 0x11550b80 rank 1 nranks 2 cudaDev 1 nvmlDev 5 busId 8e000 commId 0xb7d48819bedd2d1b - Init COMPLETE n136-128-154:773515:774460 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.70 (kernels 0.11, alloc 0.39, bootstrap 0.00, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.02, rest 0.11) n136-128-154:773514:774483 [0] NCCL INFO Channel 00/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:773514:774483 [0] NCCL INFO Channel 01/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:773514:774483 [0] NCCL INFO Channel 02/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:773515:774484 [1] NCCL INFO Channel 00/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773514:774483 [0] NCCL INFO Channel 03/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:773515:774484 [1] NCCL INFO Channel 01/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773514:774483 [0] NCCL INFO Channel 04/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:773515:774484 [1] NCCL INFO Channel 02/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773514:774483 [0] NCCL INFO Channel 05/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:773515:774484 [1] NCCL INFO Channel 03/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773514:774483 [0] NCCL INFO Channel 06/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:773515:774484 [1] NCCL INFO Channel 04/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773514:774483 [0] NCCL INFO Channel 07/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:773515:774484 [1] NCCL INFO Channel 05/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773514:774483 [0] NCCL INFO Channel 08/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:773515:774484 [1] NCCL INFO Channel 06/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:774484 [1] NCCL INFO Channel 07/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773514:774483 [0] NCCL INFO Channel 09/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:773515:774484 [1] NCCL INFO Channel 08/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773514:774483 [0] NCCL INFO Channel 10/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:773515:774484 [1] NCCL INFO Channel 09/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773514:774483 [0] NCCL INFO Channel 11/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:773515:774484 [1] NCCL INFO Channel 10/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773514:774483 [0] NCCL INFO Channel 12/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:773515:774484 [1] NCCL INFO Channel 11/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773514:774483 [0] NCCL INFO Channel 13/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:773515:774484 [1] NCCL INFO Channel 12/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773514:774483 [0] NCCL INFO Channel 14/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:773515:774484 [1] NCCL INFO Channel 13/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773514:774483 [0] NCCL INFO Channel 15/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:773515:774484 [1] NCCL INFO Channel 14/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773514:774483 [0] NCCL INFO Channel 16/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:773515:774484 [1] NCCL INFO Channel 15/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773514:774483 [0] NCCL INFO Channel 17/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:773515:774484 [1] NCCL INFO Channel 16/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773514:774483 [0] NCCL INFO Channel 18/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:773515:774484 [1] NCCL INFO Channel 17/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773514:774483 [0] NCCL INFO Channel 19/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:773515:774484 [1] NCCL INFO Channel 18/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773514:774483 [0] NCCL INFO Channel 20/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:773515:774484 [1] NCCL INFO Channel 19/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773514:774483 [0] NCCL INFO Channel 21/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:773515:774484 [1] NCCL INFO Channel 20/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773514:774483 [0] NCCL INFO Channel 22/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:773515:774484 [1] NCCL INFO Channel 21/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773514:774483 [0] NCCL INFO Channel 23/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:773515:774484 [1] NCCL INFO Channel 22/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:774484 [1] NCCL INFO Channel 23/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773514:774483 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 n136-128-154:773515:774484 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 2025-12-01:19:47:03 INFO [evaluator:559] Running generate_until requests 2025-12-01:19:47:03 INFO [evaluator:559] Running generate_until requests Passed argument batch_size = auto. Detecting largest batch size Running generate_until requests: 0%| | 0/8972 [00:00 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 01/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 02/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 03/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 04/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 05/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 06/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 07/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 08/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 09/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 10/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 11/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 12/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 13/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 14/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 15/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 16/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 17/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 18/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 19/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 20/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 21/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 22/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 23/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 24/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 25/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 26/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 27/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 28/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 29/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 30/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:773515:783690 [1] NCCL INFO Channel 31/1 : 1[5] -> 0[4] via P2P/CUMEM/read fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization' To add an exception for this directory, call: git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization n136-128-154:773515:783721 [1] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:773515:783721 [1] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:773515:783721 [1] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:773515:783721 [1] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:773515:783721 [1] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:773515:783721 [1] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:773515:774479 [1] NCCL INFO misc/socket.cc:915 -> 3 2025-12-01:20:17:29 INFO [loggers.evaluation_tracker:209] Saving results aggregated hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-14B), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto | Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr| |--------|------:|-----------------|-----:|-----------|---|-----:|---|-----:| |triviaqa| 3|remove_whitespace| 0|exact_match|↑ |0.5946|± |0.0037| [rank0]:[W1201 20:17:32.980379540 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) n136-128-154:773514:783791 [0] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:773514:783791 [0] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:773514:783791 [0] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:773514:774481 [0] NCCL INFO misc/socket.cc:915 -> 3 n136-128-154:773514:783791 [0] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:773514:783791 [0] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:773514:783791 [0] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:773515:774479 [1] NCCL INFO misc/socket.cc:915 -> 3 n136-128-154:773515:783721 [1] NCCL INFO comm 0x11550b80 rank 1 nranks 2 cudaDev 1 busId 8e000 - Abort COMPLETE n136-128-154:773514:783791 [0] NCCL INFO comm 0x1135d5c0 rank 0 nranks 2 cudaDev 0 busId 89000 - Abort COMPLETE 任务 triviaqa 评估完成! ================================================== 开始评估:任务=winogrande | 少样本数=5 | 模型=Qwen2.5-14B 输出路径:results2/Qwen2.5-14B/base_winogrande.json ================================================== The following values were not passed to `accelerate launch` and had defaults used instead: More than one GPU was found, enabling multi-GPU training. If this was unintended please pass in `--num_processes=1`. `--num_machines` was set to a value of `1` `--mixed_precision` was set to a value of `'no'` `--dynamo_backend` was set to a value of `'no'` To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`. 2025-12-01:20:18:25 INFO [__main__:440] Selected Tasks: ['winogrande'] 2025-12-01:20:18:25 INFO [__main__:440] Selected Tasks: ['winogrande'] 2025-12-01:20:18:25 INFO [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2025-12-01:20:18:25 INFO [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-14B'} 2025-12-01:20:18:25 INFO [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2025-12-01:20:18:25 INFO [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-14B'} 2025-12-01:20:18:26 WARNING [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. 2025-12-01:20:18:26 INFO [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'} 2025-12-01:20:18:26 INFO [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'} `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! Loading checkpoint shards: 0%| | 0/8 [00:00 n136-128-154:783896:784345 [0] NCCL INFO Initialized NET plugin IB n136-128-154:783896:784345 [0] NCCL INFO Assigned NET plugin IB to comm n136-128-154:783896:784345 [0] NCCL INFO Using network IB n136-128-154:783896:784345 [0] NCCL INFO ncclCommInitRankConfig comm 0x120f9cc0 rank 0 nranks 2 cudaDev 0 nvmlDev 4 busId 89000 commId 0xb0657eeb8d61eb1a - Init START 2025-12-01:20:18:51 WARNING [evaluator:309] Overwriting default num_fewshot of winogrande from None to 5 2025-12-01:20:18:51 INFO [api.task:434] Building contexts for winogrande on rank 1... 0%| | 0/633 [00:00 n136-128-154:783897:784352 [1] NCCL INFO Initialized NET plugin IB n136-128-154:783897:784352 [1] NCCL INFO Assigned NET plugin IB to comm n136-128-154:783897:784352 [1] NCCL INFO Using network IB n136-128-154:783897:784352 [1] NCCL INFO ncclCommInitRankConfig comm 0x1026e7f0 rank 1 nranks 2 cudaDev 1 nvmlDev 5 busId 8e000 commId 0xb0657eeb8d61eb1a - Init START n136-128-154:783897:784352 [1] NCCL INFO RAS client listening socket at ::1<28028> n136-128-154:783896:784345 [0] NCCL INFO RAS client listening socket at ::1<28028> n136-128-154:783897:784352 [1] NCCL INFO TOPO/NET : Importing network plugins to topology n136-128-154:783897:784352 [1] NCCL INFO Retrieving state for IB n136-128-154:783897:784352 [1] NCCL INFO Initialized state 0 for IB n136-128-154:783897:784352 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null) n136-128-154:783896:784345 [0] NCCL INFO TOPO/NET : Importing network plugins to topology n136-128-154:783896:784345 [0] NCCL INFO Retrieving state for IB n136-128-154:783896:784345 [0] NCCL INFO Initialized state 0 for IB n136-128-154:783897:784352 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null) n136-128-154:783896:784345 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null) n136-128-154:783897:784352 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null) n136-128-154:783896:784345 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null) n136-128-154:783897:784352 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null) n136-128-154:783896:784345 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null) n136-128-154:783896:784345 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null) n136-128-154:783897:784352 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 2 (distance 5 <= 5), read 0 mode Default n136-128-154:783897:784352 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 2 (distance 5 <= 5), read 0 mode Default n136-128-154:783896:784345 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 2 (distance 5 <= 5), read 0 mode Default n136-128-154:783896:784345 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 2 (distance 5 <= 5), read 0 mode Default n136-128-154:783897:784352 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 === n136-128-154:783897:784352 [1] NCCL INFO CPU/0-1 (1/1/2) n136-128-154:783897:784352 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff) n136-128-154:783897:784352 [1] NCCL INFO + PCI[24.0] - PCI/0-87000 (1000c01010de13b8) n136-128-154:783897:784352 [1] NCCL INFO + PCI[24.0] - GPU/0-89000 (0) n136-128-154:783897:784352 [1] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:783897:784352 [1] NCCL INFO + PCI[24.0] - NIC/0-95000 n136-128-154:783897:784352 [1] NCCL INFO + PCI[24.0] - PCI/0-8c000 (1000c01010de13b8) n136-128-154:783897:784352 [1] NCCL INFO + PCI[24.0] - GPU/0-8e000 (1) n136-128-154:783896:784345 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 === n136-128-154:783897:784352 [1] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:783896:784345 [0] NCCL INFO CPU/0-1 (1/1/2) n136-128-154:783897:784352 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff) n136-128-154:783896:784345 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff) n136-128-154:783897:784352 [1] NCCL INFO + PCI[24.0] - NIC/0-cc000 n136-128-154:783896:784345 [0] NCCL INFO + PCI[24.0] - PCI/0-87000 (1000c01010de13b8) n136-128-154:783897:784352 [1] NCCL INFO + SYS[10.0] - CPU/0-0 n136-128-154:783896:784345 [0] NCCL INFO + PCI[24.0] - GPU/0-89000 (0) n136-128-154:783897:784352 [1] NCCL INFO CPU/0-0 (1/1/2) n136-128-154:783896:784345 [0] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:783897:784352 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff) n136-128-154:783896:784345 [0] NCCL INFO + PCI[24.0] - NIC/0-95000 n136-128-154:783897:784352 [1] NCCL INFO + PCI[24.0] - NIC/0-1d000 n136-128-154:783896:784345 [0] NCCL INFO + PCI[24.0] - PCI/0-8c000 (1000c01010de13b8) n136-128-154:783897:784352 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff) n136-128-154:783896:784345 [0] NCCL INFO + PCI[24.0] - GPU/0-8e000 (1) n136-128-154:783897:784352 [1] NCCL INFO + PCI[24.0] - NIC/0-60000 n136-128-154:783896:784345 [0] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:783897:784352 [1] NCCL INFO + SYS[10.0] - CPU/0-1 n136-128-154:783896:784345 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff) n136-128-154:783897:784352 [1] NCCL INFO ========================================== n136-128-154:783896:784345 [0] NCCL INFO + PCI[24.0] - NIC/0-cc000 n136-128-154:783896:784345 [0] NCCL INFO + SYS[10.0] - CPU/0-0 n136-128-154:783896:784345 [0] NCCL INFO CPU/0-0 (1/1/2) n136-128-154:783897:784352 [1] NCCL INFO GPU/0-89000 :GPU/0-89000 (0/5000.0/LOC) GPU/0-8e000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:783896:784345 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff) n136-128-154:783896:784345 [0] NCCL INFO + PCI[24.0] - NIC/0-1d000 n136-128-154:783897:784352 [1] NCCL INFO GPU/0-8e000 :GPU/0-89000 (2/240.0/NVL) GPU/0-8e000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:783896:784345 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff) n136-128-154:783896:784345 [0] NCCL INFO + PCI[24.0] - NIC/0-60000 n136-128-154:783896:784345 [0] NCCL INFO + SYS[10.0] - CPU/0-1 n136-128-154:783897:784352 [1] NCCL INFO Setting affinity for GPU 5 to 33-62,97-126 n136-128-154:783896:784345 [0] NCCL INFO ========================================== n136-128-154:783896:784345 [0] NCCL INFO GPU/0-89000 :GPU/0-89000 (0/5000.0/LOC) GPU/0-8e000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:783896:784345 [0] NCCL INFO GPU/0-8e000 :GPU/0-89000 (2/240.0/NVL) GPU/0-8e000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:783896:784345 [0] NCCL INFO Setting affinity for GPU 4 to 33-62,97-126 n136-128-154:783897:784352 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1 n136-128-154:783897:784352 [1] NCCL INFO 0 : GPU/0-89000 GPU/0-8e000 n136-128-154:783897:784352 [1] NCCL INFO 1 : GPU/0-89000 GPU/0-8e000 n136-128-154:783897:784352 [1] NCCL INFO 2 : GPU/0-89000 GPU/0-8e000 n136-128-154:783897:784352 [1] NCCL INFO 3 : GPU/0-89000 GPU/0-8e000 n136-128-154:783897:784352 [1] NCCL INFO 4 : GPU/0-89000 GPU/0-8e000 n136-128-154:783897:784352 [1] NCCL INFO 5 : GPU/0-89000 GPU/0-8e000 n136-128-154:783897:784352 [1] NCCL INFO 6 : GPU/0-89000 GPU/0-8e000 n136-128-154:783897:784352 [1] NCCL INFO 7 : GPU/0-89000 GPU/0-8e000 n136-128-154:783897:784352 [1] NCCL INFO 8 : GPU/0-89000 GPU/0-8e000 n136-128-154:783897:784352 [1] NCCL INFO 9 : GPU/0-89000 GPU/0-8e000 n136-128-154:783897:784352 [1] NCCL INFO 10 : GPU/0-89000 GPU/0-8e000 n136-128-154:783897:784352 [1] NCCL INFO 11 : GPU/0-89000 GPU/0-8e000 n136-128-154:783897:784352 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0 n136-128-154:783897:784352 [1] NCCL INFO 0 : GPU/0-89000 GPU/0-8e000 n136-128-154:783897:784352 [1] NCCL INFO 1 : GPU/0-89000 GPU/0-8e000 n136-128-154:783897:784352 [1] NCCL INFO 2 : GPU/0-89000 GPU/0-8e000 n136-128-154:783897:784352 [1] NCCL INFO 3 : GPU/0-89000 GPU/0-8e000 n136-128-154:783897:784352 [1] NCCL INFO 4 : GPU/0-89000 GPU/0-8e000 n136-128-154:783897:784352 [1] NCCL INFO 5 : GPU/0-89000 GPU/0-8e000 n136-128-154:783897:784352 [1] NCCL INFO 6 : GPU/0-8e000 GPU/0-89000 n136-128-154:783897:784352 [1] NCCL INFO 7 : GPU/0-8e000 GPU/0-89000 n136-128-154:783897:784352 [1] NCCL INFO 8 : GPU/0-8e000 GPU/0-89000 n136-128-154:783897:784352 [1] NCCL INFO 9 : GPU/0-8e000 GPU/0-89000 n136-128-154:783897:784352 [1] NCCL INFO 10 : GPU/0-8e000 GPU/0-89000 n136-128-154:783897:784352 [1] NCCL INFO 11 : GPU/0-8e000 GPU/0-89000 n136-128-154:783896:784345 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1 n136-128-154:783896:784345 [0] NCCL INFO 0 : GPU/0-89000 GPU/0-8e000 n136-128-154:783896:784345 [0] NCCL INFO 1 : GPU/0-89000 GPU/0-8e000 n136-128-154:783896:784345 [0] NCCL INFO 2 : GPU/0-89000 GPU/0-8e000 n136-128-154:783896:784345 [0] NCCL INFO 3 : GPU/0-89000 GPU/0-8e000 n136-128-154:783896:784345 [0] NCCL INFO 4 : GPU/0-89000 GPU/0-8e000 n136-128-154:783896:784345 [0] NCCL INFO 5 : GPU/0-89000 GPU/0-8e000 n136-128-154:783896:784345 [0] NCCL INFO 6 : GPU/0-89000 GPU/0-8e000 n136-128-154:783896:784345 [0] NCCL INFO 7 : GPU/0-89000 GPU/0-8e000 n136-128-154:783896:784345 [0] NCCL INFO 8 : GPU/0-89000 GPU/0-8e000 n136-128-154:783896:784345 [0] NCCL INFO 9 : GPU/0-89000 GPU/0-8e000 n136-128-154:783896:784345 [0] NCCL INFO 10 : GPU/0-89000 GPU/0-8e000 n136-128-154:783896:784345 [0] NCCL INFO 11 : GPU/0-89000 GPU/0-8e000 n136-128-154:783896:784345 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0 n136-128-154:783896:784345 [0] NCCL INFO 0 : GPU/0-89000 GPU/0-8e000 n136-128-154:783896:784345 [0] NCCL INFO 1 : GPU/0-89000 GPU/0-8e000 n136-128-154:783896:784345 [0] NCCL INFO 2 : GPU/0-89000 GPU/0-8e000 n136-128-154:783896:784345 [0] NCCL INFO 3 : GPU/0-89000 GPU/0-8e000 n136-128-154:783896:784345 [0] NCCL INFO 4 : GPU/0-89000 GPU/0-8e000 n136-128-154:783896:784345 [0] NCCL INFO 5 : GPU/0-89000 GPU/0-8e000 n136-128-154:783896:784345 [0] NCCL INFO 6 : GPU/0-8e000 GPU/0-89000 n136-128-154:783896:784345 [0] NCCL INFO 7 : GPU/0-8e000 GPU/0-89000 n136-128-154:783896:784345 [0] NCCL INFO 8 : GPU/0-8e000 GPU/0-89000 n136-128-154:783896:784345 [0] NCCL INFO 9 : GPU/0-8e000 GPU/0-89000 n136-128-154:783896:784345 [0] NCCL INFO 10 : GPU/0-8e000 GPU/0-89000 n136-128-154:783896:784345 [0] NCCL INFO 11 : GPU/0-8e000 GPU/0-89000 n136-128-154:783897:784352 [1] NCCL INFO comm 0x1026e7f0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0 n136-128-154:783896:784345 [0] NCCL INFO comm 0x120f9cc0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0 n136-128-154:783897:784352 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1 n136-128-154:783897:784352 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1 n136-128-154:783897:784352 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1 n136-128-154:783897:784352 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1 n136-128-154:783897:784352 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1 n136-128-154:783897:784352 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1 n136-128-154:783897:784352 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1 n136-128-154:783897:784352 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1 n136-128-154:783897:784352 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1 n136-128-154:783897:784352 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1 n136-128-154:783897:784352 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1 n136-128-154:783897:784352 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1 n136-128-154:783897:784352 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1 n136-128-154:783897:784352 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1 n136-128-154:783897:784352 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1 n136-128-154:783897:784352 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1 n136-128-154:783897:784352 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1 n136-128-154:783897:784352 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1 n136-128-154:783897:784352 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1 n136-128-154:783897:784352 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1 n136-128-154:783897:784352 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1 n136-128-154:783897:784352 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1 n136-128-154:783897:784352 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1 n136-128-154:783897:784352 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1 n136-128-154:783896:784345 [0] NCCL INFO Channel 00/24 : 0 1 n136-128-154:783897:784352 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0 n136-128-154:783896:784345 [0] NCCL INFO Channel 01/24 : 0 1 n136-128-154:783897:784352 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0 n136-128-154:783896:784345 [0] NCCL INFO Channel 02/24 : 0 1 n136-128-154:783897:784352 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0 n136-128-154:783896:784345 [0] NCCL INFO Channel 03/24 : 0 1 n136-128-154:783897:784352 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0 n136-128-154:783896:784345 [0] NCCL INFO Channel 04/24 : 0 1 n136-128-154:783897:784352 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0 n136-128-154:783896:784345 [0] NCCL INFO Channel 05/24 : 0 1 n136-128-154:783897:784352 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0 n136-128-154:783896:784345 [0] NCCL INFO Channel 06/24 : 0 1 n136-128-154:783897:784352 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0 n136-128-154:783896:784345 [0] NCCL INFO Channel 07/24 : 0 1 n136-128-154:783897:784352 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0 n136-128-154:783896:784345 [0] NCCL INFO Channel 08/24 : 0 1 n136-128-154:783897:784352 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0 n136-128-154:783896:784345 [0] NCCL INFO Channel 09/24 : 0 1 n136-128-154:783897:784352 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0 n136-128-154:783896:784345 [0] NCCL INFO Channel 10/24 : 0 1 n136-128-154:783897:784352 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0 n136-128-154:783896:784345 [0] NCCL INFO Channel 11/24 : 0 1 n136-128-154:783897:784352 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0 n136-128-154:783896:784345 [0] NCCL INFO Channel 12/24 : 0 1 n136-128-154:783897:784352 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0 n136-128-154:783896:784345 [0] NCCL INFO Channel 13/24 : 0 1 n136-128-154:783897:784352 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0 n136-128-154:783896:784345 [0] NCCL INFO Channel 14/24 : 0 1 n136-128-154:783897:784352 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0 n136-128-154:783896:784345 [0] NCCL INFO Channel 15/24 : 0 1 n136-128-154:783897:784352 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0 n136-128-154:783896:784345 [0] NCCL INFO Channel 16/24 : 0 1 n136-128-154:783897:784352 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0 n136-128-154:783896:784345 [0] NCCL INFO Channel 17/24 : 0 1 n136-128-154:783897:784352 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0 n136-128-154:783896:784345 [0] NCCL INFO Channel 18/24 : 0 1 n136-128-154:783897:784352 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0 n136-128-154:783896:784345 [0] NCCL INFO Channel 19/24 : 0 1 n136-128-154:783897:784352 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0 n136-128-154:783896:784345 [0] NCCL INFO Channel 20/24 : 0 1 n136-128-154:783897:784352 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0 n136-128-154:783896:784345 [0] NCCL INFO Channel 21/24 : 0 1 n136-128-154:783897:784352 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0 n136-128-154:783896:784345 [0] NCCL INFO Channel 22/24 : 0 1 n136-128-154:783897:784352 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0 n136-128-154:783896:784345 [0] NCCL INFO Channel 23/24 : 0 1 n136-128-154:783897:784352 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0 n136-128-154:783897:784352 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1 n136-128-154:783896:784345 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1 n136-128-154:783896:784345 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1 n136-128-154:783897:784352 [1] NCCL INFO P2P Chunksize set to 524288 n136-128-154:783896:784345 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1 n136-128-154:783896:784345 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1 n136-128-154:783896:784345 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1 n136-128-154:783896:784345 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1 n136-128-154:783896:784345 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1 n136-128-154:783896:784345 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1 n136-128-154:783896:784345 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1 n136-128-154:783896:784345 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1 n136-128-154:783896:784345 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1 n136-128-154:783896:784345 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1 n136-128-154:783896:784345 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1 n136-128-154:783896:784345 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1 n136-128-154:783896:784345 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1 n136-128-154:783896:784345 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1 n136-128-154:783896:784345 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1 n136-128-154:783896:784345 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1 n136-128-154:783896:784345 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1 n136-128-154:783896:784345 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1 n136-128-154:783896:784345 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1 n136-128-154:783896:784345 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1 n136-128-154:783896:784345 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1 n136-128-154:783896:784345 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1 n136-128-154:783896:784345 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1 n136-128-154:783896:784345 [0] NCCL INFO P2P Chunksize set to 524288 n136-128-154:783897:784352 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. n136-128-154:783897:784359 [1] NCCL INFO [Proxy Service] Device 1 CPU core 56 n136-128-154:783897:784360 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 125 n136-128-154:783896:784345 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. n136-128-154:783896:784345 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 n136-128-154:783896:784361 [0] NCCL INFO [Proxy Service] Device 0 CPU core 101 n136-128-154:783896:784362 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 103 n136-128-154:783897:784352 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 n136-128-154:783897:784352 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer n136-128-154:783896:784345 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 n136-128-154:783896:784345 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer n136-128-154:783896:784345 [0] NCCL INFO CC Off, workFifoBytes 1048576 n136-128-154:783897:784352 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. n136-128-154:783897:784352 [1] NCCL INFO ncclCommInitRankConfig comm 0x1026e7f0 rank 1 nranks 2 cudaDev 1 nvmlDev 5 busId 8e000 commId 0xb0657eeb8d61eb1a - Init COMPLETE n136-128-154:783897:784352 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.43 (kernels 0.11, alloc 0.11, bootstrap 0.01, allgathers 0.01, topo 0.07, graphs 0.00, connections 0.06, rest 0.05) n136-128-154:783896:784345 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. n136-128-154:783896:784345 [0] NCCL INFO ncclCommInitRankConfig comm 0x120f9cc0 rank 0 nranks 2 cudaDev 0 nvmlDev 4 busId 89000 commId 0xb0657eeb8d61eb1a - Init COMPLETE n136-128-154:783896:784345 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.88 (kernels 0.13, alloc 0.14, bootstrap 0.42, allgathers 0.01, topo 0.07, graphs 0.00, connections 0.06, rest 0.05) n136-128-154:783897:784363 [1] NCCL INFO Channel 00/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784363 [1] NCCL INFO Channel 01/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784363 [1] NCCL INFO Channel 02/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784363 [1] NCCL INFO Channel 03/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784363 [1] NCCL INFO Channel 04/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784363 [1] NCCL INFO Channel 05/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784363 [1] NCCL INFO Channel 06/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784363 [1] NCCL INFO Channel 07/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Channel 00/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:783897:784363 [1] NCCL INFO Channel 08/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Channel 01/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:783897:784363 [1] NCCL INFO Channel 09/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Channel 02/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:783897:784363 [1] NCCL INFO Channel 10/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Channel 03/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:783897:784363 [1] NCCL INFO Channel 11/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Channel 04/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:783897:784363 [1] NCCL INFO Channel 12/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784363 [1] NCCL INFO Channel 13/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Channel 05/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:783897:784363 [1] NCCL INFO Channel 14/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Channel 06/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:783897:784363 [1] NCCL INFO Channel 15/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Channel 07/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:783897:784363 [1] NCCL INFO Channel 16/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Channel 08/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:783897:784363 [1] NCCL INFO Channel 17/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Channel 09/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:783897:784363 [1] NCCL INFO Channel 18/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Channel 10/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:783897:784363 [1] NCCL INFO Channel 19/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Channel 11/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:783897:784363 [1] NCCL INFO Channel 20/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Channel 12/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:783897:784363 [1] NCCL INFO Channel 21/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Channel 13/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:783897:784363 [1] NCCL INFO Channel 22/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Channel 14/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:783897:784363 [1] NCCL INFO Channel 23/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Channel 15/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Channel 16/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Channel 17/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Channel 18/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Channel 19/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Channel 20/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Channel 21/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Channel 22/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Channel 23/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:783896:784364 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 n136-128-154:783897:784363 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 2025-12-01:20:18:51 INFO [evaluator:559] Running loglikelihood requests 2025-12-01:20:18:51 INFO [evaluator:559] Running loglikelihood requests Running loglikelihood requests: 0%| | 0/1268 [00:00 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 01/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 02/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 03/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 04/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 05/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 06/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 07/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 08/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 09/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 10/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 11/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 12/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 13/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 14/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 15/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 16/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 17/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 18/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 19/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 20/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 21/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 22/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 23/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 24/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 25/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 26/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 27/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 28/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 29/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 30/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:783897:784551 [1] NCCL INFO Channel 31/1 : 1[5] -> 0[4] via P2P/CUMEM/read fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization' To add an exception for this directory, call: git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization n136-128-154:783897:784558 [1] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:783897:784558 [1] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:783897:784558 [1] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:783897:784558 [1] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:783897:784558 [1] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:783897:784558 [1] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:783897:784359 [1] NCCL INFO misc/socket.cc:915 -> 3 2025-12-01:20:19:39 INFO [loggers.evaluation_tracker:209] Saving results aggregated hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-14B), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: auto (64) | Tasks |Version|Filter|n-shot|Metric| |Value | |Stderr| |----------|------:|------|-----:|------|---|-----:|---|-----:| |winogrande| 1|none | 5|acc |↑ |0.8027|± |0.0112| [rank0]:[W1201 20:19:40.798219156 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) n136-128-154:783896:784638 [0] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:783896:784638 [0] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:783896:784638 [0] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:783896:784361 [0] NCCL INFO misc/socket.cc:915 -> 3 n136-128-154:783896:784638 [0] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:783896:784638 [0] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:783896:784638 [0] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:783897:784359 [1] NCCL INFO misc/socket.cc:915 -> 3 n136-128-154:783897:784558 [1] NCCL INFO comm 0x1026e7f0 rank 1 nranks 2 cudaDev 1 busId 8e000 - Abort COMPLETE n136-128-154:783896:784638 [0] NCCL INFO comm 0x120f9cc0 rank 0 nranks 2 cudaDev 0 busId 89000 - Abort COMPLETE 任务 winogrande 评估完成! ================================================== 开始评估:任务=mmlu | 少样本数=5 | 模型=Qwen2.5-14B 输出路径:results2/Qwen2.5-14B/base_mmlu.json ================================================== The following values were not passed to `accelerate launch` and had defaults used instead: More than one GPU was found, enabling multi-GPU training. If this was unintended please pass in `--num_processes=1`. `--num_machines` was set to a value of `1` `--mixed_precision` was set to a value of `'no'` `--dynamo_backend` was set to a value of `'no'` To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`. 2025-12-01:20:20:30 INFO [__main__:440] Selected Tasks: ['mmlu'] 2025-12-01:20:20:30 INFO [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2025-12-01:20:20:30 INFO [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-14B'} 2025-12-01:20:20:30 INFO [__main__:440] Selected Tasks: ['mmlu'] 2025-12-01:20:20:30 INFO [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2025-12-01:20:20:30 INFO [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-14B'} 2025-12-01:20:20:31 WARNING [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. 2025-12-01:20:20:31 INFO [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'} `torch_dtype` is deprecated! Use `dtype` instead! Loading checkpoint shards: 0%| | 0/8 [00:00 n136-128-154:784738:789648 [0] NCCL INFO Initialized NET plugin IB n136-128-154:784738:789648 [0] NCCL INFO Assigned NET plugin IB to comm n136-128-154:784738:789648 [0] NCCL INFO Using network IB n136-128-154:784738:789648 [0] NCCL INFO ncclCommInitRankConfig comm 0xfcd17b0 rank 0 nranks 2 cudaDev 0 nvmlDev 4 busId 89000 commId 0x2b676ada57c74a0c - Init START 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_abstract_algebra from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_anatomy from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_astronomy from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_college_biology from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_college_chemistry from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_college_computer_science from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_college_mathematics from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_college_physics from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_computer_security from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_conceptual_physics from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_electrical_engineering from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_elementary_mathematics from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_high_school_biology from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_high_school_chemistry from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_high_school_computer_science from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_high_school_mathematics from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_high_school_physics from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_high_school_statistics from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_machine_learning from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_business_ethics from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_clinical_knowledge from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_college_medicine from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_global_facts from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_human_aging from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_management from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_marketing from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_medical_genetics from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_miscellaneous from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_nutrition from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_professional_accounting from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_professional_medicine from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_virology from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_econometrics from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_high_school_geography from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_high_school_government_and_politics from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_high_school_macroeconomics from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_high_school_microeconomics from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_high_school_psychology from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_human_sexuality from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_professional_psychology from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_public_relations from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_security_studies from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_sociology from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_us_foreign_policy from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_formal_logic from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_high_school_european_history from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_high_school_us_history from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_high_school_world_history from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_international_law from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_jurisprudence from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_logical_fallacies from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_moral_disputes from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_moral_scenarios from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_philosophy from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_prehistory from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_professional_law from None to 5 2025-12-01:20:37:50 WARNING [evaluator:309] Overwriting default num_fewshot of mmlu_world_religions from None to 5 2025-12-01:20:37:50 INFO [api.task:434] Building contexts for mmlu_abstract_algebra on rank 1... 0%| | 0/50 [00:00 n136-128-154:784739:789983 [1] NCCL INFO Initialized NET plugin IB n136-128-154:784739:789983 [1] NCCL INFO Assigned NET plugin IB to comm n136-128-154:784739:789983 [1] NCCL INFO Using network IB n136-128-154:784739:789983 [1] NCCL INFO ncclCommInitRankConfig comm 0x112a0cd0 rank 1 nranks 2 cudaDev 1 nvmlDev 5 busId 8e000 commId 0x2b676ada57c74a0c - Init START n136-128-154:784739:789983 [1] NCCL INFO RAS client listening socket at ::1<28028> n136-128-154:784738:789648 [0] NCCL INFO RAS client listening socket at ::1<28028> n136-128-154:784738:789648 [0] NCCL INFO TOPO/NET : Importing network plugins to topology n136-128-154:784738:789648 [0] NCCL INFO Retrieving state for IB n136-128-154:784738:789648 [0] NCCL INFO Initialized state 0 for IB n136-128-154:784738:789648 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null) n136-128-154:784739:789983 [1] NCCL INFO TOPO/NET : Importing network plugins to topology n136-128-154:784739:789983 [1] NCCL INFO Retrieving state for IB n136-128-154:784739:789983 [1] NCCL INFO Initialized state 0 for IB n136-128-154:784738:789648 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null) n136-128-154:784739:789983 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null) n136-128-154:784738:789648 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null) n136-128-154:784739:789983 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null) n136-128-154:784739:789983 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null) n136-128-154:784738:789648 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null) n136-128-154:784739:789983 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null) n136-128-154:784739:789983 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 2 (distance 5 <= 5), read 0 mode Default n136-128-154:784739:789983 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 2 (distance 5 <= 5), read 0 mode Default n136-128-154:784738:789648 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 2 (distance 5 <= 5), read 0 mode Default n136-128-154:784738:789648 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 2 (distance 5 <= 5), read 0 mode Default n136-128-154:784739:789983 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 === n136-128-154:784739:789983 [1] NCCL INFO CPU/0-1 (1/1/2) n136-128-154:784739:789983 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff) n136-128-154:784739:789983 [1] NCCL INFO + PCI[24.0] - PCI/0-87000 (1000c01010de13b8) n136-128-154:784739:789983 [1] NCCL INFO + PCI[24.0] - GPU/0-89000 (0) n136-128-154:784739:789983 [1] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:784739:789983 [1] NCCL INFO + PCI[24.0] - NIC/0-95000 n136-128-154:784739:789983 [1] NCCL INFO + PCI[24.0] - PCI/0-8c000 (1000c01010de13b8) n136-128-154:784739:789983 [1] NCCL INFO + PCI[24.0] - GPU/0-8e000 (1) n136-128-154:784739:789983 [1] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:784739:789983 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff) n136-128-154:784739:789983 [1] NCCL INFO + PCI[24.0] - NIC/0-cc000 n136-128-154:784739:789983 [1] NCCL INFO + SYS[10.0] - CPU/0-0 n136-128-154:784739:789983 [1] NCCL INFO CPU/0-0 (1/1/2) n136-128-154:784739:789983 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff) n136-128-154:784739:789983 [1] NCCL INFO + PCI[24.0] - NIC/0-1d000 n136-128-154:784739:789983 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff) n136-128-154:784739:789983 [1] NCCL INFO + PCI[24.0] - NIC/0-60000 n136-128-154:784739:789983 [1] NCCL INFO + SYS[10.0] - CPU/0-1 n136-128-154:784739:789983 [1] NCCL INFO ========================================== n136-128-154:784739:789983 [1] NCCL INFO GPU/0-89000 :GPU/0-89000 (0/5000.0/LOC) GPU/0-8e000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:784739:789983 [1] NCCL INFO GPU/0-8e000 :GPU/0-89000 (2/240.0/NVL) GPU/0-8e000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:784739:789983 [1] NCCL INFO Setting affinity for GPU 5 to 33-62,97-126 n136-128-154:784738:789648 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 === n136-128-154:784738:789648 [0] NCCL INFO CPU/0-1 (1/1/2) n136-128-154:784738:789648 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff) n136-128-154:784738:789648 [0] NCCL INFO + PCI[24.0] - PCI/0-87000 (1000c01010de13b8) n136-128-154:784738:789648 [0] NCCL INFO + PCI[24.0] - GPU/0-89000 (0) n136-128-154:784738:789648 [0] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:784738:789648 [0] NCCL INFO + PCI[24.0] - NIC/0-95000 n136-128-154:784738:789648 [0] NCCL INFO + PCI[24.0] - PCI/0-8c000 (1000c01010de13b8) n136-128-154:784738:789648 [0] NCCL INFO + PCI[24.0] - GPU/0-8e000 (1) n136-128-154:784738:789648 [0] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:784738:789648 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff) n136-128-154:784738:789648 [0] NCCL INFO + PCI[24.0] - NIC/0-cc000 n136-128-154:784738:789648 [0] NCCL INFO + SYS[10.0] - CPU/0-0 n136-128-154:784738:789648 [0] NCCL INFO CPU/0-0 (1/1/2) n136-128-154:784738:789648 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff) n136-128-154:784738:789648 [0] NCCL INFO + PCI[24.0] - NIC/0-1d000 n136-128-154:784738:789648 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff) n136-128-154:784738:789648 [0] NCCL INFO + PCI[24.0] - NIC/0-60000 n136-128-154:784738:789648 [0] NCCL INFO + SYS[10.0] - CPU/0-1 n136-128-154:784738:789648 [0] NCCL INFO ========================================== n136-128-154:784738:789648 [0] NCCL INFO GPU/0-89000 :GPU/0-89000 (0/5000.0/LOC) GPU/0-8e000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:784738:789648 [0] NCCL INFO GPU/0-8e000 :GPU/0-89000 (2/240.0/NVL) GPU/0-8e000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:784738:789648 [0] NCCL INFO Setting affinity for GPU 4 to 33-62,97-126 n136-128-154:784738:789648 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1 n136-128-154:784738:789648 [0] NCCL INFO 0 : GPU/0-89000 GPU/0-8e000 n136-128-154:784738:789648 [0] NCCL INFO 1 : GPU/0-89000 GPU/0-8e000 n136-128-154:784738:789648 [0] NCCL INFO 2 : GPU/0-89000 GPU/0-8e000 n136-128-154:784738:789648 [0] NCCL INFO 3 : GPU/0-89000 GPU/0-8e000 n136-128-154:784738:789648 [0] NCCL INFO 4 : GPU/0-89000 GPU/0-8e000 n136-128-154:784738:789648 [0] NCCL INFO 5 : GPU/0-89000 GPU/0-8e000 n136-128-154:784738:789648 [0] NCCL INFO 6 : GPU/0-89000 GPU/0-8e000 n136-128-154:784738:789648 [0] NCCL INFO 7 : GPU/0-89000 GPU/0-8e000 n136-128-154:784738:789648 [0] NCCL INFO 8 : GPU/0-89000 GPU/0-8e000 n136-128-154:784738:789648 [0] NCCL INFO 9 : GPU/0-89000 GPU/0-8e000 n136-128-154:784738:789648 [0] NCCL INFO 10 : GPU/0-89000 GPU/0-8e000 n136-128-154:784738:789648 [0] NCCL INFO 11 : GPU/0-89000 GPU/0-8e000 n136-128-154:784738:789648 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0 n136-128-154:784738:789648 [0] NCCL INFO 0 : GPU/0-89000 GPU/0-8e000 n136-128-154:784738:789648 [0] NCCL INFO 1 : GPU/0-89000 GPU/0-8e000 n136-128-154:784738:789648 [0] NCCL INFO 2 : GPU/0-89000 GPU/0-8e000 n136-128-154:784738:789648 [0] NCCL INFO 3 : GPU/0-89000 GPU/0-8e000 n136-128-154:784738:789648 [0] NCCL INFO 4 : GPU/0-89000 GPU/0-8e000 n136-128-154:784738:789648 [0] NCCL INFO 5 : GPU/0-89000 GPU/0-8e000 n136-128-154:784738:789648 [0] NCCL INFO 6 : GPU/0-8e000 GPU/0-89000 n136-128-154:784738:789648 [0] NCCL INFO 7 : GPU/0-8e000 GPU/0-89000 n136-128-154:784738:789648 [0] NCCL INFO 8 : GPU/0-8e000 GPU/0-89000 n136-128-154:784738:789648 [0] NCCL INFO 9 : GPU/0-8e000 GPU/0-89000 n136-128-154:784738:789648 [0] NCCL INFO 10 : GPU/0-8e000 GPU/0-89000 n136-128-154:784738:789648 [0] NCCL INFO 11 : GPU/0-8e000 GPU/0-89000 n136-128-154:784739:789983 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1 n136-128-154:784739:789983 [1] NCCL INFO 0 : GPU/0-89000 GPU/0-8e000 n136-128-154:784739:789983 [1] NCCL INFO 1 : GPU/0-89000 GPU/0-8e000 n136-128-154:784739:789983 [1] NCCL INFO 2 : GPU/0-89000 GPU/0-8e000 n136-128-154:784739:789983 [1] NCCL INFO 3 : GPU/0-89000 GPU/0-8e000 n136-128-154:784739:789983 [1] NCCL INFO 4 : GPU/0-89000 GPU/0-8e000 n136-128-154:784739:789983 [1] NCCL INFO 5 : GPU/0-89000 GPU/0-8e000 n136-128-154:784739:789983 [1] NCCL INFO 6 : GPU/0-89000 GPU/0-8e000 n136-128-154:784739:789983 [1] NCCL INFO 7 : GPU/0-89000 GPU/0-8e000 n136-128-154:784739:789983 [1] NCCL INFO 8 : GPU/0-89000 GPU/0-8e000 n136-128-154:784739:789983 [1] NCCL INFO 9 : GPU/0-89000 GPU/0-8e000 n136-128-154:784739:789983 [1] NCCL INFO 10 : GPU/0-89000 GPU/0-8e000 n136-128-154:784739:789983 [1] NCCL INFO 11 : GPU/0-89000 GPU/0-8e000 n136-128-154:784739:789983 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0 n136-128-154:784739:789983 [1] NCCL INFO 0 : GPU/0-89000 GPU/0-8e000 n136-128-154:784739:789983 [1] NCCL INFO 1 : GPU/0-89000 GPU/0-8e000 n136-128-154:784739:789983 [1] NCCL INFO 2 : GPU/0-89000 GPU/0-8e000 n136-128-154:784739:789983 [1] NCCL INFO 3 : GPU/0-89000 GPU/0-8e000 n136-128-154:784739:789983 [1] NCCL INFO 4 : GPU/0-89000 GPU/0-8e000 n136-128-154:784739:789983 [1] NCCL INFO 5 : GPU/0-89000 GPU/0-8e000 n136-128-154:784739:789983 [1] NCCL INFO 6 : GPU/0-8e000 GPU/0-89000 n136-128-154:784739:789983 [1] NCCL INFO 7 : GPU/0-8e000 GPU/0-89000 n136-128-154:784739:789983 [1] NCCL INFO 8 : GPU/0-8e000 GPU/0-89000 n136-128-154:784739:789983 [1] NCCL INFO 9 : GPU/0-8e000 GPU/0-89000 n136-128-154:784739:789983 [1] NCCL INFO 10 : GPU/0-8e000 GPU/0-89000 n136-128-154:784739:789983 [1] NCCL INFO 11 : GPU/0-8e000 GPU/0-89000 n136-128-154:784738:789648 [0] NCCL INFO comm 0xfcd17b0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0 n136-128-154:784739:789983 [1] NCCL INFO comm 0x112a0cd0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0 n136-128-154:784738:789648 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1 n136-128-154:784739:789983 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1 n136-128-154:784738:789648 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1 n136-128-154:784739:789983 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1 n136-128-154:784738:789648 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1 n136-128-154:784739:789983 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1 n136-128-154:784738:789648 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1 n136-128-154:784739:789983 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1 n136-128-154:784738:789648 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1 n136-128-154:784739:789983 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1 n136-128-154:784738:789648 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1 n136-128-154:784739:789983 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1 n136-128-154:784738:789648 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1 n136-128-154:784739:789983 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1 n136-128-154:784738:789648 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1 n136-128-154:784739:789983 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1 n136-128-154:784738:789648 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1 n136-128-154:784739:789983 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1 n136-128-154:784738:789648 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1 n136-128-154:784739:789983 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1 n136-128-154:784738:789648 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1 n136-128-154:784739:789983 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1 n136-128-154:784738:789648 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1 n136-128-154:784739:789983 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1 n136-128-154:784738:789648 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1 n136-128-154:784739:789983 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1 n136-128-154:784738:789648 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1 n136-128-154:784739:789983 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1 n136-128-154:784738:789648 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1 n136-128-154:784739:789983 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1 n136-128-154:784738:789648 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1 n136-128-154:784739:789983 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1 n136-128-154:784738:789648 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1 n136-128-154:784739:789983 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1 n136-128-154:784738:789648 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1 n136-128-154:784739:789983 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1 n136-128-154:784738:789648 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1 n136-128-154:784739:789983 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1 n136-128-154:784738:789648 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1 n136-128-154:784739:789983 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1 n136-128-154:784738:789648 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1 n136-128-154:784739:789983 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1 n136-128-154:784738:789648 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1 n136-128-154:784739:789983 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1 n136-128-154:784738:789648 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1 n136-128-154:784739:789983 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1 n136-128-154:784738:789648 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1 n136-128-154:784739:789983 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1 n136-128-154:784738:789648 [0] NCCL INFO Channel 00/24 : 0 1 n136-128-154:784738:789648 [0] NCCL INFO Channel 01/24 : 0 1 n136-128-154:784739:789983 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0 n136-128-154:784738:789648 [0] NCCL INFO Channel 02/24 : 0 1 n136-128-154:784739:789983 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0 n136-128-154:784738:789648 [0] NCCL INFO Channel 03/24 : 0 1 n136-128-154:784739:789983 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0 n136-128-154:784738:789648 [0] NCCL INFO Channel 04/24 : 0 1 n136-128-154:784739:789983 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0 n136-128-154:784738:789648 [0] NCCL INFO Channel 05/24 : 0 1 n136-128-154:784739:789983 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0 n136-128-154:784738:789648 [0] NCCL INFO Channel 06/24 : 0 1 n136-128-154:784739:789983 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0 n136-128-154:784738:789648 [0] NCCL INFO Channel 07/24 : 0 1 n136-128-154:784739:789983 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0 n136-128-154:784738:789648 [0] NCCL INFO Channel 08/24 : 0 1 n136-128-154:784739:789983 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0 n136-128-154:784738:789648 [0] NCCL INFO Channel 09/24 : 0 1 n136-128-154:784739:789983 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0 n136-128-154:784738:789648 [0] NCCL INFO Channel 10/24 : 0 1 n136-128-154:784739:789983 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0 n136-128-154:784738:789648 [0] NCCL INFO Channel 11/24 : 0 1 n136-128-154:784739:789983 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0 n136-128-154:784738:789648 [0] NCCL INFO Channel 12/24 : 0 1 n136-128-154:784739:789983 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0 n136-128-154:784738:789648 [0] NCCL INFO Channel 13/24 : 0 1 n136-128-154:784739:789983 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0 n136-128-154:784738:789648 [0] NCCL INFO Channel 14/24 : 0 1 n136-128-154:784739:789983 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0 n136-128-154:784738:789648 [0] NCCL INFO Channel 15/24 : 0 1 n136-128-154:784739:789983 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0 n136-128-154:784738:789648 [0] NCCL INFO Channel 16/24 : 0 1 n136-128-154:784739:789983 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0 n136-128-154:784738:789648 [0] NCCL INFO Channel 17/24 : 0 1 n136-128-154:784739:789983 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0 n136-128-154:784738:789648 [0] NCCL INFO Channel 18/24 : 0 1 n136-128-154:784739:789983 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0 n136-128-154:784738:789648 [0] NCCL INFO Channel 19/24 : 0 1 n136-128-154:784739:789983 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0 n136-128-154:784738:789648 [0] NCCL INFO Channel 20/24 : 0 1 n136-128-154:784739:789983 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0 n136-128-154:784738:789648 [0] NCCL INFO Channel 21/24 : 0 1 n136-128-154:784739:789983 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0 n136-128-154:784738:789648 [0] NCCL INFO Channel 22/24 : 0 1 n136-128-154:784739:789983 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0 n136-128-154:784738:789648 [0] NCCL INFO Channel 23/24 : 0 1 n136-128-154:784739:789983 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0 n136-128-154:784738:789648 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1 n136-128-154:784739:789983 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0 n136-128-154:784739:789983 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1 n136-128-154:784738:789648 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1 n136-128-154:784738:789648 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1 n136-128-154:784739:789983 [1] NCCL INFO P2P Chunksize set to 524288 n136-128-154:784738:789648 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1 n136-128-154:784738:789648 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1 n136-128-154:784738:789648 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1 n136-128-154:784738:789648 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1 n136-128-154:784738:789648 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1 n136-128-154:784738:789648 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1 n136-128-154:784738:789648 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1 n136-128-154:784738:789648 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1 n136-128-154:784738:789648 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1 n136-128-154:784738:789648 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1 n136-128-154:784738:789648 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1 n136-128-154:784738:789648 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1 n136-128-154:784738:789648 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1 n136-128-154:784738:789648 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1 n136-128-154:784738:789648 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1 n136-128-154:784738:789648 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1 n136-128-154:784738:789648 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1 n136-128-154:784738:789648 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1 n136-128-154:784738:789648 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1 n136-128-154:784738:789648 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1 n136-128-154:784738:789648 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1 n136-128-154:784738:789648 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1 n136-128-154:784738:789648 [0] NCCL INFO P2P Chunksize set to 524288 n136-128-154:784739:789983 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. n136-128-154:784739:789991 [1] NCCL INFO [Proxy Service] Device 1 CPU core 36 n136-128-154:784739:789992 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 38 n136-128-154:784738:789648 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. n136-128-154:784738:789648 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 n136-128-154:784738:789994 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 57 n136-128-154:784738:789993 [0] NCCL INFO [Proxy Service] Device 0 CPU core 54 n136-128-154:784739:789983 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 n136-128-154:784739:789983 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer n136-128-154:784738:789648 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 n136-128-154:784738:789648 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer n136-128-154:784738:789648 [0] NCCL INFO CC Off, workFifoBytes 1048576 n136-128-154:784738:789648 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. n136-128-154:784738:789648 [0] NCCL INFO ncclCommInitRankConfig comm 0xfcd17b0 rank 0 nranks 2 cudaDev 0 nvmlDev 4 busId 89000 commId 0x2b676ada57c74a0c - Init COMPLETE n136-128-154:784738:789648 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 75.79 (kernels 0.13, alloc 0.31, bootstrap 75.09, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.06, rest 0.13) n136-128-154:784739:789983 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. n136-128-154:784739:789983 [1] NCCL INFO ncclCommInitRankConfig comm 0x112a0cd0 rank 1 nranks 2 cudaDev 1 nvmlDev 5 busId 8e000 commId 0x2b676ada57c74a0c - Init COMPLETE n136-128-154:784739:789983 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.59 (kernels 0.13, alloc 0.20, bootstrap 0.00, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.06, rest 0.13) n136-128-154:784738:789995 [0] NCCL INFO Channel 00/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:784738:789995 [0] NCCL INFO Channel 01/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:784738:789995 [0] NCCL INFO Channel 02/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:784738:789995 [0] NCCL INFO Channel 03/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:784738:789995 [0] NCCL INFO Channel 04/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:784738:789995 [0] NCCL INFO Channel 05/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Channel 00/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784738:789995 [0] NCCL INFO Channel 06/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Channel 01/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784738:789995 [0] NCCL INFO Channel 07/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Channel 02/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784738:789995 [0] NCCL INFO Channel 08/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Channel 03/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784738:789995 [0] NCCL INFO Channel 09/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Channel 04/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784738:789995 [0] NCCL INFO Channel 10/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Channel 05/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784738:789995 [0] NCCL INFO Channel 11/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Channel 06/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784738:789995 [0] NCCL INFO Channel 12/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Channel 07/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784738:789995 [0] NCCL INFO Channel 13/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Channel 08/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784738:789995 [0] NCCL INFO Channel 14/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Channel 09/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784738:789995 [0] NCCL INFO Channel 15/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Channel 10/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784738:789995 [0] NCCL INFO Channel 16/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Channel 11/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784738:789995 [0] NCCL INFO Channel 17/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Channel 12/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784738:789995 [0] NCCL INFO Channel 18/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Channel 13/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784738:789995 [0] NCCL INFO Channel 19/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Channel 14/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784738:789995 [0] NCCL INFO Channel 20/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Channel 15/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784738:789995 [0] NCCL INFO Channel 21/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Channel 16/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784738:789995 [0] NCCL INFO Channel 22/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Channel 17/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784738:789995 [0] NCCL INFO Channel 23/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Channel 18/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Channel 19/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Channel 20/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Channel 21/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Channel 22/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Channel 23/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:789996 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 n136-128-154:784738:789995 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 2025-12-01:20:37:51 INFO [api.task:434] Building contexts for mmlu_anatomy on rank 0... 2025-12-01:20:37:51 INFO [api.task:434] Building contexts for mmlu_anatomy on rank 1... 0%| | 0/67 [00:00 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 01/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 02/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 03/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 04/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 05/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 06/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 07/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 08/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 09/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 10/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 11/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 12/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 13/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 14/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 15/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 16/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 17/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 18/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 19/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 20/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 21/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 22/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 23/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 24/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 25/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 26/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 27/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 28/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 29/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 30/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:784739:793819 [1] NCCL INFO Channel 31/1 : 1[5] -> 0[4] via P2P/CUMEM/read fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization' To add an exception for this directory, call: git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization n136-128-154:784739:793824 [1] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:784739:793824 [1] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:784739:793824 [1] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:784739:793824 [1] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:784739:793824 [1] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:784739:793824 [1] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:784739:789991 [1] NCCL INFO misc/socket.cc:915 -> 3 2025-12-01:20:56:24 INFO [loggers.evaluation_tracker:209] Saving results aggregated hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-14B), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: auto (13) | Tasks |Version|Filter|n-shot|Metric| |Value | |Stderr| |---------------------------------------|------:|------|-----:|------|---|-----:|---|-----:| |mmlu | 2|none | |acc |↑ |0.7978|± |0.0032| | - humanities | 2|none | |acc |↑ |0.7447|± |0.0061| | - formal_logic | 1|none | 5|acc |↑ |0.6349|± |0.0431| | - high_school_european_history | 1|none | 5|acc |↑ |0.8606|± |0.0270| | - high_school_us_history | 1|none | 5|acc |↑ |0.9167|± |0.0194| | - high_school_world_history | 1|none | 5|acc |↑ |0.9198|± |0.0177| | - international_law | 1|none | 5|acc |↑ |0.8760|± |0.0301| | - jurisprudence | 1|none | 5|acc |↑ |0.8519|± |0.0343| | - logical_fallacies | 1|none | 5|acc |↑ |0.8834|± |0.0252| | - moral_disputes | 1|none | 5|acc |↑ |0.8237|± |0.0205| | - moral_scenarios | 1|none | 5|acc |↑ |0.6939|± |0.0154| | - philosophy | 1|none | 5|acc |↑ |0.8328|± |0.0212| | - prehistory | 1|none | 5|acc |↑ |0.8827|± |0.0179| | - professional_law | 1|none | 5|acc |↑ |0.6050|± |0.0125| | - world_religions | 1|none | 5|acc |↑ |0.9123|± |0.0217| | - other | 2|none | |acc |↑ |0.8256|± |0.0066| | - business_ethics | 1|none | 5|acc |↑ |0.7800|± |0.0416| | - clinical_knowledge | 1|none | 5|acc |↑ |0.8566|± |0.0216| | - college_medicine | 1|none | 5|acc |↑ |0.7861|± |0.0313| | - global_facts | 1|none | 5|acc |↑ |0.5900|± |0.0494| | - human_aging | 1|none | 5|acc |↑ |0.7937|± |0.0272| | - management | 1|none | 5|acc |↑ |0.9029|± |0.0293| | - marketing | 1|none | 5|acc |↑ |0.9530|± |0.0139| | - medical_genetics | 1|none | 5|acc |↑ |0.9100|± |0.0288| | - miscellaneous | 1|none | 5|acc |↑ |0.9042|± |0.0105| | - nutrition | 1|none | 5|acc |↑ |0.8366|± |0.0212| | - professional_accounting | 1|none | 5|acc |↑ |0.6631|± |0.0282| | - professional_medicine | 1|none | 5|acc |↑ |0.8603|± |0.0211| | - virology | 1|none | 5|acc |↑ |0.5783|± |0.0384| | - social sciences | 2|none | |acc |↑ |0.8713|± |0.0059| | - econometrics | 1|none | 5|acc |↑ |0.6842|± |0.0437| | - high_school_geography | 1|none | 5|acc |↑ |0.9141|± |0.0200| | - high_school_government_and_politics| 1|none | 5|acc |↑ |0.9637|± |0.0135| | - high_school_macroeconomics | 1|none | 5|acc |↑ |0.8462|± |0.0183| | - high_school_microeconomics | 1|none | 5|acc |↑ |0.9328|± |0.0163| | - high_school_psychology | 1|none | 5|acc |↑ |0.9193|± |0.0117| | - human_sexuality | 1|none | 5|acc |↑ |0.8931|± |0.0271| | - professional_psychology | 1|none | 5|acc |↑ |0.8235|± |0.0154| | - public_relations | 1|none | 5|acc |↑ |0.7091|± |0.0435| | - security_studies | 1|none | 5|acc |↑ |0.8571|± |0.0224| | - sociology | 1|none | 5|acc |↑ |0.9055|± |0.0207| | - us_foreign_policy | 1|none | 5|acc |↑ |0.9200|± |0.0273| | - stem | 2|none | |acc |↑ |0.7780|± |0.0071| | - abstract_algebra | 1|none | 5|acc |↑ |0.6300|± |0.0485| | - anatomy | 1|none | 5|acc |↑ |0.7407|± |0.0379| | - astronomy | 1|none | 5|acc |↑ |0.9145|± |0.0228| | - college_biology | 1|none | 5|acc |↑ |0.9306|± |0.0213| | - college_chemistry | 1|none | 5|acc |↑ |0.5400|± |0.0501| | - college_computer_science | 1|none | 5|acc |↑ |0.7300|± |0.0446| | - college_mathematics | 1|none | 5|acc |↑ |0.5900|± |0.0494| | - college_physics | 1|none | 5|acc |↑ |0.6471|± |0.0476| | - computer_security | 1|none | 5|acc |↑ |0.7800|± |0.0416| | - conceptual_physics | 1|none | 5|acc |↑ |0.8638|± |0.0224| | - electrical_engineering | 1|none | 5|acc |↑ |0.7724|± |0.0349| | - elementary_mathematics | 1|none | 5|acc |↑ |0.8704|± |0.0173| | - high_school_biology | 1|none | 5|acc |↑ |0.9129|± |0.0160| | - high_school_chemistry | 1|none | 5|acc |↑ |0.7044|± |0.0321| | - high_school_computer_science | 1|none | 5|acc |↑ |0.9000|± |0.0302| | - high_school_mathematics | 1|none | 5|acc |↑ |0.6370|± |0.0293| | - high_school_physics | 1|none | 5|acc |↑ |0.6887|± |0.0378| | - high_school_statistics | 1|none | 5|acc |↑ |0.8056|± |0.0270| | - machine_learning | 1|none | 5|acc |↑ |0.6875|± |0.0440| | Groups |Version|Filter|n-shot|Metric| |Value | |Stderr| |------------------|------:|------|------|------|---|-----:|---|-----:| |mmlu | 2|none | |acc |↑ |0.7978|± |0.0032| | - humanities | 2|none | |acc |↑ |0.7447|± |0.0061| | - other | 2|none | |acc |↑ |0.8256|± |0.0066| | - social sciences| 2|none | |acc |↑ |0.8713|± |0.0059| | - stem | 2|none | |acc |↑ |0.7780|± |0.0071| [rank0]:[W1201 20:56:25.449196221 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) n136-128-154:784738:793874 [0] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:784738:793874 [0] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:784738:793874 [0] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:784738:789993 [0] NCCL INFO misc/socket.cc:915 -> 3 n136-128-154:784738:793874 [0] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:784738:793874 [0] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:784738:793874 [0] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:784739:789991 [1] NCCL INFO misc/socket.cc:915 -> 3 n136-128-154:784739:793824 [1] NCCL INFO comm 0x112a0cd0 rank 1 nranks 2 cudaDev 1 busId 8e000 - Abort COMPLETE n136-128-154:784738:793874 [0] NCCL INFO comm 0xfcd17b0 rank 0 nranks 2 cudaDev 0 busId 89000 - Abort COMPLETE 任务 mmlu 评估完成! ================================================== 开始评估:任务=arc_challenge | 少样本数=25 | 模型=Qwen2.5-14B 输出路径:results2/Qwen2.5-14B/base_arc_challenge.json ================================================== The following values were not passed to `accelerate launch` and had defaults used instead: More than one GPU was found, enabling multi-GPU training. If this was unintended please pass in `--num_processes=1`. `--num_machines` was set to a value of `1` `--mixed_precision` was set to a value of `'no'` `--dynamo_backend` was set to a value of `'no'` To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`. 2025-12-01:20:57:15 INFO [__main__:440] Selected Tasks: ['arc_challenge'] 2025-12-01:20:57:15 INFO [__main__:440] Selected Tasks: ['arc_challenge'] 2025-12-01:20:57:15 INFO [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2025-12-01:20:57:15 INFO [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-14B'} 2025-12-01:20:57:15 INFO [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2025-12-01:20:57:15 INFO [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-14B'} 2025-12-01:20:57:17 WARNING [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. 2025-12-01:20:57:17 INFO [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'} 2025-12-01:20:57:17 INFO [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'} `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! Loading checkpoint shards: 0%| | 0/8 [00:00 n136-128-154:793919:794583 [0] NCCL INFO Initialized NET plugin IB n136-128-154:793920:794584 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. n136-128-154:793920:794584 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0> n136-128-154:793920:794584 [1] NCCL INFO Initialized NET plugin IB n136-128-154:793919:794583 [0] NCCL INFO Assigned NET plugin IB to comm n136-128-154:793919:794583 [0] NCCL INFO Using network IB n136-128-154:793920:794584 [1] NCCL INFO Assigned NET plugin IB to comm n136-128-154:793920:794584 [1] NCCL INFO Using network IB n136-128-154:793920:794584 [1] NCCL INFO ncclCommInitRankConfig comm 0x12fe9780 rank 1 nranks 2 cudaDev 1 nvmlDev 5 busId 8e000 commId 0x369c4dd9c618f891 - Init START n136-128-154:793919:794583 [0] NCCL INFO ncclCommInitRankConfig comm 0xfc17f80 rank 0 nranks 2 cudaDev 0 nvmlDev 4 busId 89000 commId 0x369c4dd9c618f891 - Init START n136-128-154:793920:794584 [1] NCCL INFO RAS client listening socket at ::1<28028> n136-128-154:793919:794583 [0] NCCL INFO RAS client listening socket at ::1<28028> n136-128-154:793919:794583 [0] NCCL INFO TOPO/NET : Importing network plugins to topology n136-128-154:793919:794583 [0] NCCL INFO Retrieving state for IB n136-128-154:793919:794583 [0] NCCL INFO Initialized state 0 for IB n136-128-154:793919:794583 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null) n136-128-154:793920:794584 [1] NCCL INFO TOPO/NET : Importing network plugins to topology n136-128-154:793920:794584 [1] NCCL INFO Retrieving state for IB n136-128-154:793920:794584 [1] NCCL INFO Initialized state 0 for IB n136-128-154:793919:794583 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null) n136-128-154:793920:794584 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null) n136-128-154:793919:794583 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null) n136-128-154:793920:794584 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null) n136-128-154:793919:794583 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null) n136-128-154:793920:794584 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null) n136-128-154:793920:794584 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null) n136-128-154:793919:794583 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 2 (distance 5 <= 5), read 0 mode Default n136-128-154:793919:794583 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 2 (distance 5 <= 5), read 0 mode Default n136-128-154:793920:794584 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 2 (distance 5 <= 5), read 0 mode Default n136-128-154:793920:794584 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 2 (distance 5 <= 5), read 0 mode Default n136-128-154:793919:794583 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 === n136-128-154:793919:794583 [0] NCCL INFO CPU/0-1 (1/1/2) n136-128-154:793919:794583 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff) n136-128-154:793919:794583 [0] NCCL INFO + PCI[24.0] - PCI/0-87000 (1000c01010de13b8) n136-128-154:793919:794583 [0] NCCL INFO + PCI[24.0] - GPU/0-89000 (0) n136-128-154:793919:794583 [0] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:793919:794583 [0] NCCL INFO + PCI[24.0] - NIC/0-95000 n136-128-154:793919:794583 [0] NCCL INFO + PCI[24.0] - PCI/0-8c000 (1000c01010de13b8) n136-128-154:793919:794583 [0] NCCL INFO + PCI[24.0] - GPU/0-8e000 (1) n136-128-154:793919:794583 [0] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:793919:794583 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff) n136-128-154:793919:794583 [0] NCCL INFO + PCI[24.0] - NIC/0-cc000 n136-128-154:793919:794583 [0] NCCL INFO + SYS[10.0] - CPU/0-0 n136-128-154:793919:794583 [0] NCCL INFO CPU/0-0 (1/1/2) n136-128-154:793919:794583 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff) n136-128-154:793919:794583 [0] NCCL INFO + PCI[24.0] - NIC/0-1d000 n136-128-154:793919:794583 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff) n136-128-154:793919:794583 [0] NCCL INFO + PCI[24.0] - NIC/0-60000 n136-128-154:793919:794583 [0] NCCL INFO + SYS[10.0] - CPU/0-1 n136-128-154:793919:794583 [0] NCCL INFO ========================================== n136-128-154:793919:794583 [0] NCCL INFO GPU/0-89000 :GPU/0-89000 (0/5000.0/LOC) GPU/0-8e000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:793919:794583 [0] NCCL INFO GPU/0-8e000 :GPU/0-89000 (2/240.0/NVL) GPU/0-8e000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:793919:794583 [0] NCCL INFO Setting affinity for GPU 4 to 33-62,97-126 n136-128-154:793920:794584 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 === n136-128-154:793920:794584 [1] NCCL INFO CPU/0-1 (1/1/2) n136-128-154:793920:794584 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff) n136-128-154:793920:794584 [1] NCCL INFO + PCI[24.0] - PCI/0-87000 (1000c01010de13b8) n136-128-154:793920:794584 [1] NCCL INFO + PCI[24.0] - GPU/0-89000 (0) n136-128-154:793920:794584 [1] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:793920:794584 [1] NCCL INFO + PCI[24.0] - NIC/0-95000 n136-128-154:793920:794584 [1] NCCL INFO + PCI[24.0] - PCI/0-8c000 (1000c01010de13b8) n136-128-154:793920:794584 [1] NCCL INFO + PCI[24.0] - GPU/0-8e000 (1) n136-128-154:793920:794584 [1] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:793920:794584 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff) n136-128-154:793920:794584 [1] NCCL INFO + PCI[24.0] - NIC/0-cc000 n136-128-154:793920:794584 [1] NCCL INFO + SYS[10.0] - CPU/0-0 n136-128-154:793920:794584 [1] NCCL INFO CPU/0-0 (1/1/2) n136-128-154:793920:794584 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff) n136-128-154:793920:794584 [1] NCCL INFO + PCI[24.0] - NIC/0-1d000 n136-128-154:793920:794584 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff) n136-128-154:793920:794584 [1] NCCL INFO + PCI[24.0] - NIC/0-60000 n136-128-154:793920:794584 [1] NCCL INFO + SYS[10.0] - CPU/0-1 n136-128-154:793920:794584 [1] NCCL INFO ========================================== n136-128-154:793920:794584 [1] NCCL INFO GPU/0-89000 :GPU/0-89000 (0/5000.0/LOC) GPU/0-8e000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:793920:794584 [1] NCCL INFO GPU/0-8e000 :GPU/0-89000 (2/240.0/NVL) GPU/0-8e000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:793920:794584 [1] NCCL INFO Setting affinity for GPU 5 to 33-62,97-126 n136-128-154:793919:794583 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1 n136-128-154:793919:794583 [0] NCCL INFO 0 : GPU/0-89000 GPU/0-8e000 n136-128-154:793919:794583 [0] NCCL INFO 1 : GPU/0-89000 GPU/0-8e000 n136-128-154:793919:794583 [0] NCCL INFO 2 : GPU/0-89000 GPU/0-8e000 n136-128-154:793919:794583 [0] NCCL INFO 3 : GPU/0-89000 GPU/0-8e000 n136-128-154:793919:794583 [0] NCCL INFO 4 : GPU/0-89000 GPU/0-8e000 n136-128-154:793919:794583 [0] NCCL INFO 5 : GPU/0-89000 GPU/0-8e000 n136-128-154:793919:794583 [0] NCCL INFO 6 : GPU/0-89000 GPU/0-8e000 n136-128-154:793919:794583 [0] NCCL INFO 7 : GPU/0-89000 GPU/0-8e000 n136-128-154:793919:794583 [0] NCCL INFO 8 : GPU/0-89000 GPU/0-8e000 n136-128-154:793919:794583 [0] NCCL INFO 9 : GPU/0-89000 GPU/0-8e000 n136-128-154:793919:794583 [0] NCCL INFO 10 : GPU/0-89000 GPU/0-8e000 n136-128-154:793919:794583 [0] NCCL INFO 11 : GPU/0-89000 GPU/0-8e000 n136-128-154:793919:794583 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0 n136-128-154:793919:794583 [0] NCCL INFO 0 : GPU/0-89000 GPU/0-8e000 n136-128-154:793919:794583 [0] NCCL INFO 1 : GPU/0-89000 GPU/0-8e000 n136-128-154:793919:794583 [0] NCCL INFO 2 : GPU/0-89000 GPU/0-8e000 n136-128-154:793919:794583 [0] NCCL INFO 3 : GPU/0-89000 GPU/0-8e000 n136-128-154:793919:794583 [0] NCCL INFO 4 : GPU/0-89000 GPU/0-8e000 n136-128-154:793919:794583 [0] NCCL INFO 5 : GPU/0-89000 GPU/0-8e000 n136-128-154:793919:794583 [0] NCCL INFO 6 : GPU/0-8e000 GPU/0-89000 n136-128-154:793919:794583 [0] NCCL INFO 7 : GPU/0-8e000 GPU/0-89000 n136-128-154:793919:794583 [0] NCCL INFO 8 : GPU/0-8e000 GPU/0-89000 n136-128-154:793919:794583 [0] NCCL INFO 9 : GPU/0-8e000 GPU/0-89000 n136-128-154:793919:794583 [0] NCCL INFO 10 : GPU/0-8e000 GPU/0-89000 n136-128-154:793919:794583 [0] NCCL INFO 11 : GPU/0-8e000 GPU/0-89000 n136-128-154:793920:794584 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1 n136-128-154:793920:794584 [1] NCCL INFO 0 : GPU/0-89000 GPU/0-8e000 n136-128-154:793920:794584 [1] NCCL INFO 1 : GPU/0-89000 GPU/0-8e000 n136-128-154:793920:794584 [1] NCCL INFO 2 : GPU/0-89000 GPU/0-8e000 n136-128-154:793920:794584 [1] NCCL INFO 3 : GPU/0-89000 GPU/0-8e000 n136-128-154:793920:794584 [1] NCCL INFO 4 : GPU/0-89000 GPU/0-8e000 n136-128-154:793920:794584 [1] NCCL INFO 5 : GPU/0-89000 GPU/0-8e000 n136-128-154:793920:794584 [1] NCCL INFO 6 : GPU/0-89000 GPU/0-8e000 n136-128-154:793920:794584 [1] NCCL INFO 7 : GPU/0-89000 GPU/0-8e000 n136-128-154:793920:794584 [1] NCCL INFO 8 : GPU/0-89000 GPU/0-8e000 n136-128-154:793920:794584 [1] NCCL INFO 9 : GPU/0-89000 GPU/0-8e000 n136-128-154:793920:794584 [1] NCCL INFO 10 : GPU/0-89000 GPU/0-8e000 n136-128-154:793920:794584 [1] NCCL INFO 11 : GPU/0-89000 GPU/0-8e000 n136-128-154:793920:794584 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0 n136-128-154:793920:794584 [1] NCCL INFO 0 : GPU/0-89000 GPU/0-8e000 n136-128-154:793920:794584 [1] NCCL INFO 1 : GPU/0-89000 GPU/0-8e000 n136-128-154:793920:794584 [1] NCCL INFO 2 : GPU/0-89000 GPU/0-8e000 n136-128-154:793920:794584 [1] NCCL INFO 3 : GPU/0-89000 GPU/0-8e000 n136-128-154:793920:794584 [1] NCCL INFO 4 : GPU/0-89000 GPU/0-8e000 n136-128-154:793920:794584 [1] NCCL INFO 5 : GPU/0-89000 GPU/0-8e000 n136-128-154:793920:794584 [1] NCCL INFO 6 : GPU/0-8e000 GPU/0-89000 n136-128-154:793920:794584 [1] NCCL INFO 7 : GPU/0-8e000 GPU/0-89000 n136-128-154:793920:794584 [1] NCCL INFO 8 : GPU/0-8e000 GPU/0-89000 n136-128-154:793920:794584 [1] NCCL INFO 9 : GPU/0-8e000 GPU/0-89000 n136-128-154:793920:794584 [1] NCCL INFO 10 : GPU/0-8e000 GPU/0-89000 n136-128-154:793920:794584 [1] NCCL INFO 11 : GPU/0-8e000 GPU/0-89000 n136-128-154:793919:794583 [0] NCCL INFO comm 0xfc17f80 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0 n136-128-154:793920:794584 [1] NCCL INFO comm 0x12fe9780 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0 n136-128-154:793919:794583 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1 n136-128-154:793919:794583 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1 n136-128-154:793919:794583 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1 n136-128-154:793920:794584 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1 n136-128-154:793919:794583 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1 n136-128-154:793920:794584 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1 n136-128-154:793919:794583 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1 n136-128-154:793920:794584 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1 n136-128-154:793919:794583 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1 n136-128-154:793920:794584 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1 n136-128-154:793919:794583 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1 n136-128-154:793920:794584 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1 n136-128-154:793919:794583 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1 n136-128-154:793920:794584 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1 n136-128-154:793919:794583 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1 n136-128-154:793920:794584 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1 n136-128-154:793919:794583 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1 n136-128-154:793920:794584 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1 n136-128-154:793919:794583 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1 n136-128-154:793920:794584 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1 n136-128-154:793919:794583 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1 n136-128-154:793920:794584 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1 n136-128-154:793919:794583 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1 n136-128-154:793920:794584 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1 n136-128-154:793919:794583 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1 n136-128-154:793920:794584 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1 n136-128-154:793919:794583 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1 n136-128-154:793920:794584 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1 n136-128-154:793919:794583 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1 n136-128-154:793920:794584 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1 n136-128-154:793919:794583 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1 n136-128-154:793920:794584 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1 n136-128-154:793919:794583 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1 n136-128-154:793920:794584 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1 n136-128-154:793919:794583 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1 n136-128-154:793920:794584 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1 n136-128-154:793919:794583 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1 n136-128-154:793920:794584 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1 n136-128-154:793919:794583 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1 n136-128-154:793920:794584 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1 n136-128-154:793919:794583 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1 n136-128-154:793920:794584 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1 n136-128-154:793919:794583 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1 n136-128-154:793920:794584 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1 n136-128-154:793919:794583 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1 n136-128-154:793920:794584 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1 n136-128-154:793920:794584 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1 n136-128-154:793920:794584 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1 n136-128-154:793919:794583 [0] NCCL INFO Channel 00/24 : 0 1 n136-128-154:793919:794583 [0] NCCL INFO Channel 01/24 : 0 1 n136-128-154:793919:794583 [0] NCCL INFO Channel 02/24 : 0 1 n136-128-154:793919:794583 [0] NCCL INFO Channel 03/24 : 0 1 n136-128-154:793919:794583 [0] NCCL INFO Channel 04/24 : 0 1 n136-128-154:793920:794584 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0 n136-128-154:793919:794583 [0] NCCL INFO Channel 05/24 : 0 1 n136-128-154:793920:794584 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0 n136-128-154:793919:794583 [0] NCCL INFO Channel 06/24 : 0 1 n136-128-154:793920:794584 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0 n136-128-154:793919:794583 [0] NCCL INFO Channel 07/24 : 0 1 n136-128-154:793920:794584 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0 n136-128-154:793919:794583 [0] NCCL INFO Channel 08/24 : 0 1 n136-128-154:793920:794584 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0 n136-128-154:793919:794583 [0] NCCL INFO Channel 09/24 : 0 1 n136-128-154:793920:794584 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0 n136-128-154:793919:794583 [0] NCCL INFO Channel 10/24 : 0 1 n136-128-154:793920:794584 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0 n136-128-154:793919:794583 [0] NCCL INFO Channel 11/24 : 0 1 n136-128-154:793920:794584 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0 n136-128-154:793919:794583 [0] NCCL INFO Channel 12/24 : 0 1 n136-128-154:793920:794584 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0 n136-128-154:793919:794583 [0] NCCL INFO Channel 13/24 : 0 1 n136-128-154:793920:794584 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0 n136-128-154:793919:794583 [0] NCCL INFO Channel 14/24 : 0 1 n136-128-154:793920:794584 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0 n136-128-154:793919:794583 [0] NCCL INFO Channel 15/24 : 0 1 n136-128-154:793920:794584 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0 n136-128-154:793919:794583 [0] NCCL INFO Channel 16/24 : 0 1 n136-128-154:793920:794584 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0 n136-128-154:793919:794583 [0] NCCL INFO Channel 17/24 : 0 1 n136-128-154:793920:794584 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0 n136-128-154:793919:794583 [0] NCCL INFO Channel 18/24 : 0 1 n136-128-154:793920:794584 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0 n136-128-154:793919:794583 [0] NCCL INFO Channel 19/24 : 0 1 n136-128-154:793920:794584 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0 n136-128-154:793919:794583 [0] NCCL INFO Channel 20/24 : 0 1 n136-128-154:793920:794584 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0 n136-128-154:793919:794583 [0] NCCL INFO Channel 21/24 : 0 1 n136-128-154:793920:794584 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0 n136-128-154:793919:794583 [0] NCCL INFO Channel 22/24 : 0 1 n136-128-154:793920:794584 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0 n136-128-154:793919:794583 [0] NCCL INFO Channel 23/24 : 0 1 n136-128-154:793920:794584 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0 n136-128-154:793919:794583 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1 n136-128-154:793920:794584 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0 n136-128-154:793919:794583 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1 n136-128-154:793920:794584 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0 n136-128-154:793919:794583 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1 n136-128-154:793920:794584 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0 n136-128-154:793919:794583 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1 n136-128-154:793920:794584 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0 n136-128-154:793919:794583 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1 n136-128-154:793920:794584 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1 n136-128-154:793919:794583 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1 n136-128-154:793919:794583 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1 n136-128-154:793920:794584 [1] NCCL INFO P2P Chunksize set to 524288 n136-128-154:793919:794583 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1 n136-128-154:793919:794583 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1 n136-128-154:793919:794583 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1 n136-128-154:793919:794583 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1 n136-128-154:793919:794583 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1 n136-128-154:793919:794583 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1 n136-128-154:793919:794583 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1 n136-128-154:793919:794583 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1 n136-128-154:793919:794583 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1 n136-128-154:793919:794583 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1 n136-128-154:793919:794583 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1 n136-128-154:793919:794583 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1 n136-128-154:793919:794583 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1 n136-128-154:793919:794583 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1 n136-128-154:793919:794583 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1 n136-128-154:793919:794583 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1 n136-128-154:793919:794583 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1 n136-128-154:793919:794583 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1 n136-128-154:793919:794583 [0] NCCL INFO P2P Chunksize set to 524288 n136-128-154:793919:794583 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. n136-128-154:793919:794583 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 n136-128-154:793919:794608 [0] NCCL INFO [Proxy Service] Device 0 CPU core 44 n136-128-154:793919:794609 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 120 n136-128-154:793920:794584 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. n136-128-154:793920:794610 [1] NCCL INFO [Proxy Service] Device 1 CPU core 45 n136-128-154:793920:794611 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 110 n136-128-154:793920:794584 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 n136-128-154:793920:794584 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer n136-128-154:793919:794583 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 n136-128-154:793919:794583 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer n136-128-154:793919:794583 [0] NCCL INFO CC Off, workFifoBytes 1048576 n136-128-154:793920:794584 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. n136-128-154:793920:794584 [1] NCCL INFO ncclCommInitRankConfig comm 0x12fe9780 rank 1 nranks 2 cudaDev 1 nvmlDev 5 busId 8e000 commId 0x369c4dd9c618f891 - Init COMPLETE n136-128-154:793920:794584 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.80 (kernels 0.14, alloc 0.38, bootstrap 0.00, allgathers 0.00, topo 0.07, graphs 0.00, connections 0.07, rest 0.14) n136-128-154:793919:794583 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. n136-128-154:793919:794583 [0] NCCL INFO ncclCommInitRankConfig comm 0xfc17f80 rank 0 nranks 2 cudaDev 0 nvmlDev 4 busId 89000 commId 0x369c4dd9c618f891 - Init COMPLETE n136-128-154:793919:794583 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.80 (kernels 0.13, alloc 0.39, bootstrap 0.00, allgathers 0.00, topo 0.07, graphs 0.00, connections 0.07, rest 0.14) n136-128-154:793919:794613 [0] NCCL INFO Channel 00/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:793920:794612 [1] NCCL INFO Channel 00/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793919:794613 [0] NCCL INFO Channel 01/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:793920:794612 [1] NCCL INFO Channel 01/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793919:794613 [0] NCCL INFO Channel 02/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:793920:794612 [1] NCCL INFO Channel 02/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793919:794613 [0] NCCL INFO Channel 03/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:793920:794612 [1] NCCL INFO Channel 03/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793919:794613 [0] NCCL INFO Channel 04/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:793920:794612 [1] NCCL INFO Channel 04/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793919:794613 [0] NCCL INFO Channel 05/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:793920:794612 [1] NCCL INFO Channel 05/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793919:794613 [0] NCCL INFO Channel 06/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:793920:794612 [1] NCCL INFO Channel 06/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793919:794613 [0] NCCL INFO Channel 07/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:793920:794612 [1] NCCL INFO Channel 07/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793919:794613 [0] NCCL INFO Channel 08/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:793920:794612 [1] NCCL INFO Channel 08/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793919:794613 [0] NCCL INFO Channel 09/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:793920:794612 [1] NCCL INFO Channel 09/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793919:794613 [0] NCCL INFO Channel 10/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:793920:794612 [1] NCCL INFO Channel 10/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793919:794613 [0] NCCL INFO Channel 11/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:793920:794612 [1] NCCL INFO Channel 11/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793919:794613 [0] NCCL INFO Channel 12/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:793920:794612 [1] NCCL INFO Channel 12/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793919:794613 [0] NCCL INFO Channel 13/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:793920:794612 [1] NCCL INFO Channel 13/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793919:794613 [0] NCCL INFO Channel 14/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:793920:794612 [1] NCCL INFO Channel 14/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793919:794613 [0] NCCL INFO Channel 15/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:793920:794612 [1] NCCL INFO Channel 15/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793919:794613 [0] NCCL INFO Channel 16/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:793920:794612 [1] NCCL INFO Channel 16/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793919:794613 [0] NCCL INFO Channel 17/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:793920:794612 [1] NCCL INFO Channel 17/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793919:794613 [0] NCCL INFO Channel 18/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:793920:794612 [1] NCCL INFO Channel 18/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793919:794613 [0] NCCL INFO Channel 19/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:793920:794612 [1] NCCL INFO Channel 19/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793919:794613 [0] NCCL INFO Channel 20/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:793920:794612 [1] NCCL INFO Channel 20/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793919:794613 [0] NCCL INFO Channel 21/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:793920:794612 [1] NCCL INFO Channel 21/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793919:794613 [0] NCCL INFO Channel 22/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:793920:794612 [1] NCCL INFO Channel 22/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793919:794613 [0] NCCL INFO Channel 23/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:793920:794612 [1] NCCL INFO Channel 23/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793919:794613 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 n136-128-154:793920:794612 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 2025-12-01:20:58:02 INFO [evaluator:559] Running loglikelihood requests 2025-12-01:20:58:02 INFO [evaluator:559] Running loglikelihood requests Running loglikelihood requests: 0%| | 0/2344 [00:00 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 01/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 02/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 03/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 04/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 05/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 06/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 07/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 08/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 09/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 10/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 11/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 12/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 13/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 14/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 15/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 16/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 17/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 18/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 19/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 20/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 21/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 22/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 23/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 24/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 25/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 26/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 27/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 28/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 29/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 30/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:793920:798527 [1] NCCL INFO Channel 31/1 : 1[5] -> 0[4] via P2P/CUMEM/read fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization' To add an exception for this directory, call: git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization n136-128-154:793920:798554 [1] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:793920:798554 [1] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:793920:798554 [1] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:793920:798554 [1] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:793920:798554 [1] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:793920:798554 [1] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:793920:794610 [1] NCCL INFO misc/socket.cc:915 -> 3 2025-12-01:21:05:29 INFO [loggers.evaluation_tracker:209] Saving results aggregated hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-14B), gen_kwargs: (None), limit: None, num_fewshot: 25, batch_size: auto (32) | Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr| |-------------|------:|------|-----:|--------|---|-----:|---|-----:| |arc_challenge| 1|none | 25|acc |↑ |0.6416|± |0.0140| | | |none | 25|acc_norm|↑ |0.6715|± |0.0137| [rank0]:[W1201 21:05:30.444889600 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) n136-128-154:793919:798715 [0] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:793919:794608 [0] NCCL INFO misc/socket.cc:915 -> 3 n136-128-154:793919:798715 [0] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:793919:798715 [0] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:793919:798715 [0] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:793919:798715 [0] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:793919:798715 [0] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:793920:794610 [1] NCCL INFO misc/socket.cc:915 -> 3 n136-128-154:793920:798554 [1] NCCL INFO comm 0x12fe9780 rank 1 nranks 2 cudaDev 1 busId 8e000 - Abort COMPLETE n136-128-154:793919:798715 [0] NCCL INFO comm 0xfc17f80 rank 0 nranks 2 cudaDev 0 busId 89000 - Abort COMPLETE 任务 arc_challenge 评估完成! ================================================== 开始评估:任务=truthfulqa_mc1 | 少样本数=0 | 模型=Qwen2.5-14B 输出路径:results2/Qwen2.5-14B/base_truthfulqa_mc1.json ================================================== The following values were not passed to `accelerate launch` and had defaults used instead: More than one GPU was found, enabling multi-GPU training. If this was unintended please pass in `--num_processes=1`. `--num_machines` was set to a value of `1` `--mixed_precision` was set to a value of `'no'` `--dynamo_backend` was set to a value of `'no'` To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`. 2025-12-01:21:06:21 INFO [__main__:440] Selected Tasks: ['truthfulqa_mc1'] 2025-12-01:21:06:21 INFO [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2025-12-01:21:06:21 INFO [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-14B'} 2025-12-01:21:06:21 INFO [__main__:440] Selected Tasks: ['truthfulqa_mc1'] 2025-12-01:21:06:21 INFO [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2025-12-01:21:06:21 INFO [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-14B'} 2025-12-01:21:06:23 WARNING [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. 2025-12-01:21:06:23 INFO [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'} 2025-12-01:21:06:23 INFO [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'} `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! Loading checkpoint shards: 0%| | 0/8 [00:00 n136-128-154:798928:799786 [0] NCCL INFO Initialized NET plugin IB n136-128-154:798928:799786 [0] NCCL INFO Assigned NET plugin IB to comm n136-128-154:798928:799786 [0] NCCL INFO Using network IB n136-128-154:798928:799786 [0] NCCL INFO ncclCommInitRankConfig comm 0x11af83c0 rank 0 nranks 2 cudaDev 0 nvmlDev 4 busId 89000 commId 0x110064ad2e7c6376 - Init START 98%|█████████▊| 400/408 [00:00<00:00, 998.46it/s] 100%|██████████| 408/408 [00:00<00:00, 994.06it/s] n136-128-154:798929:798929 [1] NCCL INFO cudaDriverVersion 12040 n136-128-154:798929:798929 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6 n136-128-154:798929:798929 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0 n136-128-154:798929:798929 [1] NCCL INFO NCCL version 2.27.5+cuda12.9 n136-128-154:798929:798929 [1] NCCL INFO Comm config Blocking set to 1 n136-128-154:798929:799791 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. n136-128-154:798929:799791 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0. n136-128-154:798929:799791 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6 n136-128-154:798929:799791 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0 n136-128-154:798929:799791 [1] NCCL INFO NCCL_IB_HCA set to mlx5 n136-128-154:798929:799791 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. n136-128-154:798929:799791 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0> n136-128-154:798929:799791 [1] NCCL INFO Initialized NET plugin IB n136-128-154:798929:799791 [1] NCCL INFO Assigned NET plugin IB to comm n136-128-154:798929:799791 [1] NCCL INFO Using network IB n136-128-154:798929:799791 [1] NCCL INFO ncclCommInitRankConfig comm 0x12c17e40 rank 1 nranks 2 cudaDev 1 nvmlDev 5 busId 8e000 commId 0x110064ad2e7c6376 - Init START n136-128-154:798929:799791 [1] NCCL INFO RAS client listening socket at ::1<28028> n136-128-154:798928:799786 [0] NCCL INFO RAS client listening socket at ::1<28028> n136-128-154:798929:799791 [1] NCCL INFO TOPO/NET : Importing network plugins to topology n136-128-154:798929:799791 [1] NCCL INFO Retrieving state for IB n136-128-154:798929:799791 [1] NCCL INFO Initialized state 0 for IB n136-128-154:798929:799791 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null) n136-128-154:798929:799791 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null) n136-128-154:798929:799791 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null) n136-128-154:798929:799791 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null) n136-128-154:798928:799786 [0] NCCL INFO TOPO/NET : Importing network plugins to topology n136-128-154:798928:799786 [0] NCCL INFO Retrieving state for IB n136-128-154:798928:799786 [0] NCCL INFO Initialized state 0 for IB n136-128-154:798928:799786 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null) n136-128-154:798928:799786 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null) n136-128-154:798928:799786 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null) n136-128-154:798928:799786 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null) n136-128-154:798929:799791 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 2 (distance 5 <= 5), read 0 mode Default n136-128-154:798928:799786 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 2 (distance 5 <= 5), read 0 mode Default n136-128-154:798929:799791 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 2 (distance 5 <= 5), read 0 mode Default n136-128-154:798928:799786 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 2 (distance 5 <= 5), read 0 mode Default n136-128-154:798929:799791 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 === n136-128-154:798928:799786 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 === n136-128-154:798929:799791 [1] NCCL INFO CPU/0-1 (1/1/2) n136-128-154:798928:799786 [0] NCCL INFO CPU/0-1 (1/1/2) n136-128-154:798929:799791 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff) n136-128-154:798928:799786 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff) n136-128-154:798929:799791 [1] NCCL INFO + PCI[24.0] - PCI/0-87000 (1000c01010de13b8) n136-128-154:798928:799786 [0] NCCL INFO + PCI[24.0] - PCI/0-87000 (1000c01010de13b8) n136-128-154:798929:799791 [1] NCCL INFO + PCI[24.0] - GPU/0-89000 (0) n136-128-154:798928:799786 [0] NCCL INFO + PCI[24.0] - GPU/0-89000 (0) n136-128-154:798929:799791 [1] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:798928:799786 [0] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:798929:799791 [1] NCCL INFO + PCI[24.0] - NIC/0-95000 n136-128-154:798928:799786 [0] NCCL INFO + PCI[24.0] - NIC/0-95000 n136-128-154:798929:799791 [1] NCCL INFO + PCI[24.0] - PCI/0-8c000 (1000c01010de13b8) n136-128-154:798928:799786 [0] NCCL INFO + PCI[24.0] - PCI/0-8c000 (1000c01010de13b8) n136-128-154:798929:799791 [1] NCCL INFO + PCI[24.0] - GPU/0-8e000 (1) n136-128-154:798928:799786 [0] NCCL INFO + PCI[24.0] - GPU/0-8e000 (1) n136-128-154:798929:799791 [1] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:798928:799786 [0] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:798929:799791 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff) n136-128-154:798928:799786 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff) n136-128-154:798929:799791 [1] NCCL INFO + PCI[24.0] - NIC/0-cc000 n136-128-154:798928:799786 [0] NCCL INFO + PCI[24.0] - NIC/0-cc000 n136-128-154:798929:799791 [1] NCCL INFO + SYS[10.0] - CPU/0-0 n136-128-154:798928:799786 [0] NCCL INFO + SYS[10.0] - CPU/0-0 n136-128-154:798929:799791 [1] NCCL INFO CPU/0-0 (1/1/2) n136-128-154:798928:799786 [0] NCCL INFO CPU/0-0 (1/1/2) n136-128-154:798929:799791 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff) n136-128-154:798928:799786 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff) n136-128-154:798929:799791 [1] NCCL INFO + PCI[24.0] - NIC/0-1d000 n136-128-154:798928:799786 [0] NCCL INFO + PCI[24.0] - NIC/0-1d000 n136-128-154:798929:799791 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff) n136-128-154:798928:799786 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff) n136-128-154:798929:799791 [1] NCCL INFO + PCI[24.0] - NIC/0-60000 n136-128-154:798928:799786 [0] NCCL INFO + PCI[24.0] - NIC/0-60000 n136-128-154:798929:799791 [1] NCCL INFO + SYS[10.0] - CPU/0-1 n136-128-154:798928:799786 [0] NCCL INFO + SYS[10.0] - CPU/0-1 n136-128-154:798929:799791 [1] NCCL INFO ========================================== n136-128-154:798928:799786 [0] NCCL INFO ========================================== n136-128-154:798929:799791 [1] NCCL INFO GPU/0-89000 :GPU/0-89000 (0/5000.0/LOC) GPU/0-8e000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:798928:799786 [0] NCCL INFO GPU/0-89000 :GPU/0-89000 (0/5000.0/LOC) GPU/0-8e000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:798929:799791 [1] NCCL INFO GPU/0-8e000 :GPU/0-89000 (2/240.0/NVL) GPU/0-8e000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:798928:799786 [0] NCCL INFO GPU/0-8e000 :GPU/0-89000 (2/240.0/NVL) GPU/0-8e000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:798929:799791 [1] NCCL INFO Setting affinity for GPU 5 to 33-62,97-126 n136-128-154:798928:799786 [0] NCCL INFO Setting affinity for GPU 4 to 33-62,97-126 n136-128-154:798928:799786 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1 n136-128-154:798929:799791 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1 n136-128-154:798928:799786 [0] NCCL INFO 0 : GPU/0-89000 GPU/0-8e000 n136-128-154:798929:799791 [1] NCCL INFO 0 : GPU/0-89000 GPU/0-8e000 n136-128-154:798928:799786 [0] NCCL INFO 1 : GPU/0-89000 GPU/0-8e000 n136-128-154:798929:799791 [1] NCCL INFO 1 : GPU/0-89000 GPU/0-8e000 n136-128-154:798928:799786 [0] NCCL INFO 2 : GPU/0-89000 GPU/0-8e000 n136-128-154:798929:799791 [1] NCCL INFO 2 : GPU/0-89000 GPU/0-8e000 n136-128-154:798928:799786 [0] NCCL INFO 3 : GPU/0-89000 GPU/0-8e000 n136-128-154:798929:799791 [1] NCCL INFO 3 : GPU/0-89000 GPU/0-8e000 n136-128-154:798928:799786 [0] NCCL INFO 4 : GPU/0-89000 GPU/0-8e000 n136-128-154:798929:799791 [1] NCCL INFO 4 : GPU/0-89000 GPU/0-8e000 n136-128-154:798928:799786 [0] NCCL INFO 5 : GPU/0-89000 GPU/0-8e000 n136-128-154:798929:799791 [1] NCCL INFO 5 : GPU/0-89000 GPU/0-8e000 n136-128-154:798928:799786 [0] NCCL INFO 6 : GPU/0-89000 GPU/0-8e000 n136-128-154:798929:799791 [1] NCCL INFO 6 : GPU/0-89000 GPU/0-8e000 n136-128-154:798928:799786 [0] NCCL INFO 7 : GPU/0-89000 GPU/0-8e000 n136-128-154:798929:799791 [1] NCCL INFO 7 : GPU/0-89000 GPU/0-8e000 n136-128-154:798928:799786 [0] NCCL INFO 8 : GPU/0-89000 GPU/0-8e000 n136-128-154:798929:799791 [1] NCCL INFO 8 : GPU/0-89000 GPU/0-8e000 n136-128-154:798928:799786 [0] NCCL INFO 9 : GPU/0-89000 GPU/0-8e000 n136-128-154:798929:799791 [1] NCCL INFO 9 : GPU/0-89000 GPU/0-8e000 n136-128-154:798928:799786 [0] NCCL INFO 10 : GPU/0-89000 GPU/0-8e000 n136-128-154:798929:799791 [1] NCCL INFO 10 : GPU/0-89000 GPU/0-8e000 n136-128-154:798928:799786 [0] NCCL INFO 11 : GPU/0-89000 GPU/0-8e000 n136-128-154:798929:799791 [1] NCCL INFO 11 : GPU/0-89000 GPU/0-8e000 n136-128-154:798929:799791 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0 n136-128-154:798928:799786 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0 n136-128-154:798929:799791 [1] NCCL INFO 0 : GPU/0-89000 GPU/0-8e000 n136-128-154:798928:799786 [0] NCCL INFO 0 : GPU/0-89000 GPU/0-8e000 n136-128-154:798929:799791 [1] NCCL INFO 1 : GPU/0-89000 GPU/0-8e000 n136-128-154:798928:799786 [0] NCCL INFO 1 : GPU/0-89000 GPU/0-8e000 n136-128-154:798929:799791 [1] NCCL INFO 2 : GPU/0-89000 GPU/0-8e000 n136-128-154:798928:799786 [0] NCCL INFO 2 : GPU/0-89000 GPU/0-8e000 n136-128-154:798929:799791 [1] NCCL INFO 3 : GPU/0-89000 GPU/0-8e000 n136-128-154:798928:799786 [0] NCCL INFO 3 : GPU/0-89000 GPU/0-8e000 n136-128-154:798929:799791 [1] NCCL INFO 4 : GPU/0-89000 GPU/0-8e000 n136-128-154:798928:799786 [0] NCCL INFO 4 : GPU/0-89000 GPU/0-8e000 n136-128-154:798929:799791 [1] NCCL INFO 5 : GPU/0-89000 GPU/0-8e000 n136-128-154:798928:799786 [0] NCCL INFO 5 : GPU/0-89000 GPU/0-8e000 n136-128-154:798929:799791 [1] NCCL INFO 6 : GPU/0-8e000 GPU/0-89000 n136-128-154:798928:799786 [0] NCCL INFO 6 : GPU/0-8e000 GPU/0-89000 n136-128-154:798929:799791 [1] NCCL INFO 7 : GPU/0-8e000 GPU/0-89000 n136-128-154:798928:799786 [0] NCCL INFO 7 : GPU/0-8e000 GPU/0-89000 n136-128-154:798929:799791 [1] NCCL INFO 8 : GPU/0-8e000 GPU/0-89000 n136-128-154:798928:799786 [0] NCCL INFO 8 : GPU/0-8e000 GPU/0-89000 n136-128-154:798929:799791 [1] NCCL INFO 9 : GPU/0-8e000 GPU/0-89000 n136-128-154:798928:799786 [0] NCCL INFO 9 : GPU/0-8e000 GPU/0-89000 n136-128-154:798929:799791 [1] NCCL INFO 10 : GPU/0-8e000 GPU/0-89000 n136-128-154:798928:799786 [0] NCCL INFO 10 : GPU/0-8e000 GPU/0-89000 n136-128-154:798929:799791 [1] NCCL INFO 11 : GPU/0-8e000 GPU/0-89000 n136-128-154:798928:799786 [0] NCCL INFO 11 : GPU/0-8e000 GPU/0-89000 n136-128-154:798929:799791 [1] NCCL INFO comm 0x12c17e40 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0 n136-128-154:798928:799786 [0] NCCL INFO comm 0x11af83c0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0 n136-128-154:798929:799791 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1 n136-128-154:798929:799791 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1 n136-128-154:798929:799791 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1 n136-128-154:798929:799791 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1 n136-128-154:798929:799791 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1 n136-128-154:798929:799791 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1 n136-128-154:798929:799791 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1 n136-128-154:798929:799791 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1 n136-128-154:798929:799791 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1 n136-128-154:798929:799791 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1 n136-128-154:798929:799791 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1 n136-128-154:798929:799791 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1 n136-128-154:798929:799791 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1 n136-128-154:798929:799791 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1 n136-128-154:798929:799791 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1 n136-128-154:798929:799791 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1 n136-128-154:798929:799791 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1 n136-128-154:798929:799791 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1 n136-128-154:798929:799791 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1 n136-128-154:798929:799791 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1 n136-128-154:798929:799791 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1 n136-128-154:798929:799791 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1 n136-128-154:798929:799791 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1 n136-128-154:798929:799791 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1 n136-128-154:798928:799786 [0] NCCL INFO Channel 00/24 : 0 1 n136-128-154:798928:799786 [0] NCCL INFO Channel 01/24 : 0 1 n136-128-154:798929:799791 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0 n136-128-154:798928:799786 [0] NCCL INFO Channel 02/24 : 0 1 n136-128-154:798929:799791 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0 n136-128-154:798928:799786 [0] NCCL INFO Channel 03/24 : 0 1 n136-128-154:798929:799791 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0 n136-128-154:798928:799786 [0] NCCL INFO Channel 04/24 : 0 1 n136-128-154:798929:799791 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0 n136-128-154:798928:799786 [0] NCCL INFO Channel 05/24 : 0 1 n136-128-154:798929:799791 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0 n136-128-154:798928:799786 [0] NCCL INFO Channel 06/24 : 0 1 n136-128-154:798929:799791 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0 n136-128-154:798928:799786 [0] NCCL INFO Channel 07/24 : 0 1 n136-128-154:798929:799791 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0 n136-128-154:798928:799786 [0] NCCL INFO Channel 08/24 : 0 1 n136-128-154:798929:799791 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0 n136-128-154:798928:799786 [0] NCCL INFO Channel 09/24 : 0 1 n136-128-154:798929:799791 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0 n136-128-154:798928:799786 [0] NCCL INFO Channel 10/24 : 0 1 n136-128-154:798929:799791 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0 n136-128-154:798928:799786 [0] NCCL INFO Channel 11/24 : 0 1 n136-128-154:798929:799791 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0 n136-128-154:798928:799786 [0] NCCL INFO Channel 12/24 : 0 1 n136-128-154:798929:799791 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0 n136-128-154:798928:799786 [0] NCCL INFO Channel 13/24 : 0 1 n136-128-154:798929:799791 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0 n136-128-154:798928:799786 [0] NCCL INFO Channel 14/24 : 0 1 n136-128-154:798929:799791 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0 n136-128-154:798928:799786 [0] NCCL INFO Channel 15/24 : 0 1 n136-128-154:798929:799791 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0 n136-128-154:798928:799786 [0] NCCL INFO Channel 16/24 : 0 1 n136-128-154:798929:799791 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0 n136-128-154:798928:799786 [0] NCCL INFO Channel 17/24 : 0 1 n136-128-154:798929:799791 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0 n136-128-154:798928:799786 [0] NCCL INFO Channel 18/24 : 0 1 n136-128-154:798929:799791 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0 n136-128-154:798928:799786 [0] NCCL INFO Channel 19/24 : 0 1 n136-128-154:798929:799791 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0 n136-128-154:798928:799786 [0] NCCL INFO Channel 20/24 : 0 1 n136-128-154:798929:799791 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0 n136-128-154:798928:799786 [0] NCCL INFO Channel 21/24 : 0 1 n136-128-154:798929:799791 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0 n136-128-154:798928:799786 [0] NCCL INFO Channel 22/24 : 0 1 n136-128-154:798929:799791 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0 n136-128-154:798928:799786 [0] NCCL INFO Channel 23/24 : 0 1 n136-128-154:798929:799791 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0 n136-128-154:798928:799786 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1 n136-128-154:798929:799791 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0 n136-128-154:798928:799786 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1 n136-128-154:798929:799791 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1 n136-128-154:798928:799786 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1 n136-128-154:798928:799786 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1 n136-128-154:798929:799791 [1] NCCL INFO P2P Chunksize set to 524288 n136-128-154:798928:799786 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1 n136-128-154:798928:799786 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1 n136-128-154:798928:799786 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1 n136-128-154:798928:799786 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1 n136-128-154:798928:799786 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1 n136-128-154:798928:799786 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1 n136-128-154:798928:799786 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1 n136-128-154:798928:799786 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1 n136-128-154:798928:799786 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1 n136-128-154:798928:799786 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1 n136-128-154:798928:799786 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1 n136-128-154:798928:799786 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1 n136-128-154:798928:799786 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1 n136-128-154:798928:799786 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1 n136-128-154:798928:799786 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1 n136-128-154:798928:799786 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1 n136-128-154:798928:799786 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1 n136-128-154:798928:799786 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1 n136-128-154:798928:799786 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1 n136-128-154:798928:799786 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1 n136-128-154:798928:799786 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1 n136-128-154:798928:799786 [0] NCCL INFO P2P Chunksize set to 524288 n136-128-154:798929:799791 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. n136-128-154:798929:799804 [1] NCCL INFO [Proxy Service] Device 1 CPU core 40 n136-128-154:798929:799805 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 105 n136-128-154:798928:799786 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. n136-128-154:798928:799786 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 n136-128-154:798928:799808 [0] NCCL INFO [Proxy Service] Device 0 CPU core 115 n136-128-154:798928:799809 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 116 n136-128-154:798929:799791 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 n136-128-154:798929:799791 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer n136-128-154:798928:799786 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 n136-128-154:798928:799786 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer n136-128-154:798928:799786 [0] NCCL INFO CC Off, workFifoBytes 1048576 n136-128-154:798929:799791 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. n136-128-154:798929:799791 [1] NCCL INFO ncclCommInitRankConfig comm 0x12c17e40 rank 1 nranks 2 cudaDev 1 nvmlDev 5 busId 8e000 commId 0x110064ad2e7c6376 - Init COMPLETE n136-128-154:798929:799791 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.55 (kernels 0.16, alloc 0.17, bootstrap 0.00, allgathers 0.00, topo 0.05, graphs 0.00, connections 0.08, rest 0.09) n136-128-154:798928:799786 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. n136-128-154:798928:799786 [0] NCCL INFO ncclCommInitRankConfig comm 0x11af83c0 rank 0 nranks 2 cudaDev 0 nvmlDev 4 busId 89000 commId 0x110064ad2e7c6376 - Init COMPLETE n136-128-154:798928:799786 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.90 (kernels 0.11, alloc 0.18, bootstrap 0.38, allgathers 0.00, topo 0.05, graphs 0.00, connections 0.08, rest 0.09) n136-128-154:798929:799825 [1] NCCL INFO Channel 00/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:799825 [1] NCCL INFO Channel 01/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:799825 [1] NCCL INFO Channel 02/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:799825 [1] NCCL INFO Channel 03/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Channel 00/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:798929:799825 [1] NCCL INFO Channel 04/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Channel 01/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:798929:799825 [1] NCCL INFO Channel 05/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Channel 02/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:798929:799825 [1] NCCL INFO Channel 06/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:799825 [1] NCCL INFO Channel 07/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Channel 03/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:798929:799825 [1] NCCL INFO Channel 08/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Channel 04/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:798929:799825 [1] NCCL INFO Channel 09/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Channel 05/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:798929:799825 [1] NCCL INFO Channel 10/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Channel 06/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:798929:799825 [1] NCCL INFO Channel 11/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Channel 07/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:798929:799825 [1] NCCL INFO Channel 12/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:799825 [1] NCCL INFO Channel 13/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Channel 08/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:798929:799825 [1] NCCL INFO Channel 14/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Channel 09/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:798929:799825 [1] NCCL INFO Channel 15/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Channel 10/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:798929:799825 [1] NCCL INFO Channel 16/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Channel 11/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:798929:799825 [1] NCCL INFO Channel 17/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Channel 12/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:798929:799825 [1] NCCL INFO Channel 18/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Channel 13/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:798929:799825 [1] NCCL INFO Channel 19/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:799825 [1] NCCL INFO Channel 20/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Channel 14/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:798929:799825 [1] NCCL INFO Channel 21/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Channel 15/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:798929:799825 [1] NCCL INFO Channel 22/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Channel 16/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:798929:799825 [1] NCCL INFO Channel 23/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Channel 17/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Channel 18/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Channel 19/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Channel 20/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Channel 21/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Channel 22/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Channel 23/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:798928:799826 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 n136-128-154:798929:799825 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 2025-12-01:21:06:54 INFO [evaluator:559] Running loglikelihood requests 2025-12-01:21:06:54 INFO [evaluator:559] Running loglikelihood requests Running loglikelihood requests: 0%| | 0/2066 [00:00 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 01/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 02/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 03/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 04/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 05/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 06/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 07/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 08/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 09/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 10/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 11/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 12/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 13/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 14/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 15/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 16/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 17/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 18/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 19/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 20/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 21/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 22/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 23/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 24/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 25/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 26/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 27/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 28/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 29/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 30/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:798929:800765 [1] NCCL INFO Channel 31/1 : 1[5] -> 0[4] via P2P/CUMEM/read fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization' To add an exception for this directory, call: git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization n136-128-154:798929:800780 [1] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:798929:800780 [1] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:798929:800780 [1] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:798929:800780 [1] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:798929:800780 [1] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:798929:800780 [1] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:798929:799804 [1] NCCL INFO misc/socket.cc:915 -> 3 2025-12-01:21:08:19 INFO [loggers.evaluation_tracker:209] Saving results aggregated hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-14B), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto (64) | Tasks |Version|Filter|n-shot|Metric| |Value | |Stderr| |--------------|------:|------|-----:|------|---|-----:|---|-----:| |truthfulqa_mc1| 2|none | 0|acc |↑ |0.4027|± |0.0172| [rank0]:[W1201 21:08:20.327174909 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) n136-128-154:798928:801053 [0] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:798928:801053 [0] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:798928:801053 [0] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:798928:799808 [0] NCCL INFO misc/socket.cc:915 -> 3 n136-128-154:798928:801053 [0] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:798928:801053 [0] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:798928:801053 [0] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:798929:799804 [1] NCCL INFO misc/socket.cc:915 -> 3 n136-128-154:798929:800780 [1] NCCL INFO comm 0x12c17e40 rank 1 nranks 2 cudaDev 1 busId 8e000 - Abort COMPLETE n136-128-154:798928:801053 [0] NCCL INFO comm 0x11af83c0 rank 0 nranks 2 cudaDev 0 busId 89000 - Abort COMPLETE 任务 truthfulqa_mc1 评估完成! ================================================== 开始评估:任务=hellaswag | 少样本数=10 | 模型=Qwen2.5-14B 输出路径:results2/Qwen2.5-14B/base_hellaswag.json ================================================== The following values were not passed to `accelerate launch` and had defaults used instead: More than one GPU was found, enabling multi-GPU training. If this was unintended please pass in `--num_processes=1`. `--num_machines` was set to a value of `1` `--mixed_precision` was set to a value of `'no'` `--dynamo_backend` was set to a value of `'no'` To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`. 2025-12-01:21:09:11 INFO [__main__:440] Selected Tasks: ['hellaswag'] 2025-12-01:21:09:11 INFO [__main__:440] Selected Tasks: ['hellaswag'] 2025-12-01:21:09:11 INFO [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2025-12-01:21:09:11 INFO [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-14B'} 2025-12-01:21:09:11 INFO [evaluator:189] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 2025-12-01:21:09:11 INFO [evaluator:227] Initializing hf model, with arguments: {'pretrained': '/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-14B'} 2025-12-01:21:09:12 WARNING [accelerate.utils.other:513] Detected kernel version 5.4.143, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. 2025-12-01:21:09:13 INFO [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'} 2025-12-01:21:09:13 INFO [models.huggingface:382] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'} `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! Loading checkpoint shards: 0%| | 0/8 [00:00 n136-128-154:801240:802613 [0] NCCL INFO Initialized NET plugin IB n136-128-154:801240:802613 [0] NCCL INFO Assigned NET plugin IB to comm n136-128-154:801240:802613 [0] NCCL INFO Using network IB 87%|████████▋ | 4347/5021 [00:21<00:03, 201.05it/s]n136-128-154:801240:802613 [0] NCCL INFO ncclCommInitRankConfig comm 0x3e4064c0 rank 0 nranks 2 cudaDev 0 nvmlDev 4 busId 89000 commId 0xd6174eb4c702f7 - Init START 87%|████████▋ | 4368/5021 [00:22<00:03, 201.19it/s] 87%|████████▋ | 4389/5021 [00:22<00:03, 201.66it/s] 88%|████████▊ | 4410/5021 [00:22<00:03, 202.06it/s] 88%|████████▊ | 4431/5021 [00:22<00:02, 202.66it/s] 89%|████████▊ | 4452/5021 [00:22<00:02, 202.22it/s] 89%|████████▉ | 4473/5021 [00:22<00:02, 202.26it/s] 90%|████████▉ | 4494/5021 [00:22<00:02, 202.18it/s] 90%|████████▉ | 4515/5021 [00:22<00:02, 202.35it/s] 90%|█████████ | 4536/5021 [00:22<00:02, 202.41it/s] 91%|█████████ | 4557/5021 [00:22<00:02, 202.31it/s] 91%|█████████ | 4578/5021 [00:23<00:02, 202.73it/s] 92%|█████████▏| 4599/5021 [00:23<00:02, 202.52it/s] 92%|█████████▏| 4620/5021 [00:23<00:01, 202.29it/s] 92%|█████████▏| 4641/5021 [00:23<00:01, 202.16it/s] 93%|█████████▎| 4662/5021 [00:23<00:01, 201.93it/s] 93%|█████████▎| 4683/5021 [00:23<00:01, 201.78it/s] 94%|█████████▎| 4704/5021 [00:23<00:01, 201.21it/s] 94%|█████████▍| 4725/5021 [00:23<00:01, 201.15it/s] 95%|█████████▍| 4746/5021 [00:23<00:01, 201.27it/s] 95%|█████████▍| 4767/5021 [00:23<00:01, 201.53it/s] 95%|█████████▌| 4788/5021 [00:24<00:01, 201.50it/s] 96%|█████████▌| 4809/5021 [00:24<00:01, 201.87it/s] 96%|█████████▌| 4830/5021 [00:24<00:00, 201.99it/s] 97%|█████████▋| 4851/5021 [00:24<00:00, 202.62it/s] 97%|█████████▋| 4872/5021 [00:24<00:00, 202.91it/s] 97%|█████████▋| 4893/5021 [00:24<00:00, 202.65it/s] 98%|█████████▊| 4914/5021 [00:24<00:00, 201.92it/s] 98%|█████████▊| 4935/5021 [00:24<00:00, 201.75it/s] 99%|█████████▊| 4956/5021 [00:24<00:00, 200.98it/s] 99%|█████████▉| 4977/5021 [00:25<00:00, 201.14it/s] 100%|█████████▉| 4998/5021 [00:25<00:00, 201.43it/s] 100%|█████████▉| 5019/5021 [00:25<00:00, 201.78it/s] 100%|██████████| 5021/5021 [00:25<00:00, 198.81it/s] n136-128-154:801241:801241 [1] NCCL INFO cudaDriverVersion 12040 n136-128-154:801241:801241 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6 n136-128-154:801241:801241 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0 n136-128-154:801241:801241 [1] NCCL INFO NCCL version 2.27.5+cuda12.9 n136-128-154:801241:801241 [1] NCCL INFO Comm config Blocking set to 1 n136-128-154:801241:802665 [1] NCCL INFO NET/Plugin: Could not find: none libnccl-net-none.so. n136-128-154:801241:802665 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0. n136-128-154:801241:802665 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6 n136-128-154:801241:802665 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0 n136-128-154:801241:802665 [1] NCCL INFO NCCL_IB_HCA set to mlx5 n136-128-154:801241:802665 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. n136-128-154:801241:802665 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [RO]; OOB eth0:fdbd:dc03:9:451::154<0> n136-128-154:801241:802665 [1] NCCL INFO Initialized NET plugin IB n136-128-154:801241:802665 [1] NCCL INFO Assigned NET plugin IB to comm n136-128-154:801241:802665 [1] NCCL INFO Using network IB n136-128-154:801241:802665 [1] NCCL INFO ncclCommInitRankConfig comm 0x11f84b90 rank 1 nranks 2 cudaDev 1 nvmlDev 5 busId 8e000 commId 0xd6174eb4c702f7 - Init START n136-128-154:801241:802665 [1] NCCL INFO RAS client listening socket at ::1<28028> n136-128-154:801240:802613 [0] NCCL INFO RAS client listening socket at ::1<28028> n136-128-154:801240:802613 [0] NCCL INFO TOPO/NET : Importing network plugins to topology n136-128-154:801240:802613 [0] NCCL INFO Retrieving state for IB n136-128-154:801240:802613 [0] NCCL INFO Initialized state 0 for IB n136-128-154:801240:802613 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null) n136-128-154:801241:802665 [1] NCCL INFO TOPO/NET : Importing network plugins to topology n136-128-154:801241:802665 [1] NCCL INFO Retrieving state for IB n136-128-154:801241:802665 [1] NCCL INFO Initialized state 0 for IB n136-128-154:801240:802613 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null) n136-128-154:801241:802665 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_0 in topo with pciPath=/sys/devices/pci0000:09/0000:09:02.0/0000:0a:00.0/0000:0b:08.0/0000:1b:00.0/0000:1c:00.0/0000:1d:00.0 keep=1 coll=(null) n136-128-154:801240:802613 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null) n136-128-154:801241:802665 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_1 in topo with pciPath=/sys/devices/pci0000:43/0000:43:02.0/0000:44:00.0/0000:45:08.0/0000:5e:00.0/0000:5f:00.0/0000:60:00.0 keep=1 coll=(null) n136-128-154:801240:802613 [0] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null) n136-128-154:801241:802665 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_2 in topo with pciPath=/sys/devices/pci0000:82/0000:82:02.0/0000:83:00.0/0000:84:08.0/0000:93:00.0/0000:94:00.0/0000:95:00.0 keep=1 coll=(null) n136-128-154:801241:802665 [1] NCCL INFO ncclTopoPopulateNics : Filled mlx5_3 in topo with pciPath=/sys/devices/pci0000:be/0000:be:02.0/0000:bf:00.0/0000:c0:04.0/0000:ca:00.0/0000:cb:10.0/0000:cc:00.0 keep=1 coll=(null) n136-128-154:801241:802665 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 2 (distance 5 <= 5), read 0 mode Default n136-128-154:801241:802665 [1] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 2 (distance 5 <= 5), read 0 mode Default n136-128-154:801240:802613 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 0 / HCA 2 (distance 5 <= 5), read 0 mode Default n136-128-154:801240:802613 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 1 / HCA 2 (distance 5 <= 5), read 0 mode Default n136-128-154:801241:802665 [1] NCCL INFO === System : maxBw 240.0 totalBw 240.0 === n136-128-154:801241:802665 [1] NCCL INFO CPU/0-1 (1/1/2) n136-128-154:801241:802665 [1] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff) n136-128-154:801241:802665 [1] NCCL INFO + PCI[24.0] - PCI/0-87000 (1000c01010de13b8) n136-128-154:801241:802665 [1] NCCL INFO + PCI[24.0] - GPU/0-89000 (0) n136-128-154:801241:802665 [1] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:801241:802665 [1] NCCL INFO + PCI[24.0] - NIC/0-95000 n136-128-154:801241:802665 [1] NCCL INFO + PCI[24.0] - PCI/0-8c000 (1000c01010de13b8) n136-128-154:801241:802665 [1] NCCL INFO + PCI[24.0] - GPU/0-8e000 (1) n136-128-154:801241:802665 [1] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:801241:802665 [1] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff) n136-128-154:801241:802665 [1] NCCL INFO + PCI[24.0] - NIC/0-cc000 n136-128-154:801241:802665 [1] NCCL INFO + SYS[10.0] - CPU/0-0 n136-128-154:801241:802665 [1] NCCL INFO CPU/0-0 (1/1/2) n136-128-154:801241:802665 [1] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff) n136-128-154:801241:802665 [1] NCCL INFO + PCI[24.0] - NIC/0-1d000 n136-128-154:801241:802665 [1] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff) n136-128-154:801241:802665 [1] NCCL INFO + PCI[24.0] - NIC/0-60000 n136-128-154:801241:802665 [1] NCCL INFO + SYS[10.0] - CPU/0-1 n136-128-154:801240:802613 [0] NCCL INFO === System : maxBw 240.0 totalBw 240.0 === n136-128-154:801241:802665 [1] NCCL INFO ========================================== n136-128-154:801240:802613 [0] NCCL INFO CPU/0-1 (1/1/2) n136-128-154:801240:802613 [0] NCCL INFO + PCI[24.0] - PCI/0-83000 (1000c0101000ffff) n136-128-154:801241:802665 [1] NCCL INFO GPU/0-89000 :GPU/0-89000 (0/5000.0/LOC) GPU/0-8e000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:801240:802613 [0] NCCL INFO + PCI[24.0] - PCI/0-87000 (1000c01010de13b8) n136-128-154:801240:802613 [0] NCCL INFO + PCI[24.0] - GPU/0-89000 (0) n136-128-154:801241:802665 [1] NCCL INFO GPU/0-8e000 :GPU/0-89000 (2/240.0/NVL) GPU/0-8e000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:801240:802613 [0] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:801240:802613 [0] NCCL INFO + PCI[24.0] - NIC/0-95000 n136-128-154:801240:802613 [0] NCCL INFO + PCI[24.0] - PCI/0-8c000 (1000c01010de13b8) n136-128-154:801240:802613 [0] NCCL INFO + PCI[24.0] - GPU/0-8e000 (1) n136-128-154:801241:802665 [1] NCCL INFO Setting affinity for GPU 5 to 33-62,97-126 n136-128-154:801240:802613 [0] NCCL INFO + NVL[240.0] - NVS/0-0 n136-128-154:801240:802613 [0] NCCL INFO + PCI[24.0] - PCI/0-bf000 (1000c0101000ffff) n136-128-154:801240:802613 [0] NCCL INFO + PCI[24.0] - NIC/0-cc000 n136-128-154:801240:802613 [0] NCCL INFO + SYS[10.0] - CPU/0-0 n136-128-154:801240:802613 [0] NCCL INFO CPU/0-0 (1/1/2) n136-128-154:801240:802613 [0] NCCL INFO + PCI[24.0] - PCI/0-a000 (1000c0101000ffff) n136-128-154:801240:802613 [0] NCCL INFO + PCI[24.0] - NIC/0-1d000 n136-128-154:801240:802613 [0] NCCL INFO + PCI[24.0] - PCI/0-44000 (1000c0101000ffff) n136-128-154:801240:802613 [0] NCCL INFO + PCI[24.0] - NIC/0-60000 n136-128-154:801240:802613 [0] NCCL INFO + SYS[10.0] - CPU/0-1 n136-128-154:801240:802613 [0] NCCL INFO ========================================== n136-128-154:801240:802613 [0] NCCL INFO GPU/0-89000 :GPU/0-89000 (0/5000.0/LOC) GPU/0-8e000 (2/240.0/NVL) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:801240:802613 [0] NCCL INFO GPU/0-8e000 :GPU/0-89000 (2/240.0/NVL) GPU/0-8e000 (0/5000.0/LOC) NVS/0-0 (1/240.0/NVL) CPU/0-1 (3/24.0/PHB) CPU/0-0 (4/10.0/SYS) n136-128-154:801240:802613 [0] NCCL INFO Setting affinity for GPU 4 to 33-62,97-126 n136-128-154:801240:802613 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1 n136-128-154:801240:802613 [0] NCCL INFO 0 : GPU/0-89000 GPU/0-8e000 n136-128-154:801240:802613 [0] NCCL INFO 1 : GPU/0-89000 GPU/0-8e000 n136-128-154:801240:802613 [0] NCCL INFO 2 : GPU/0-89000 GPU/0-8e000 n136-128-154:801240:802613 [0] NCCL INFO 3 : GPU/0-89000 GPU/0-8e000 n136-128-154:801240:802613 [0] NCCL INFO 4 : GPU/0-89000 GPU/0-8e000 n136-128-154:801240:802613 [0] NCCL INFO 5 : GPU/0-89000 GPU/0-8e000 n136-128-154:801240:802613 [0] NCCL INFO 6 : GPU/0-89000 GPU/0-8e000 n136-128-154:801240:802613 [0] NCCL INFO 7 : GPU/0-89000 GPU/0-8e000 n136-128-154:801240:802613 [0] NCCL INFO 8 : GPU/0-89000 GPU/0-8e000 n136-128-154:801240:802613 [0] NCCL INFO 9 : GPU/0-89000 GPU/0-8e000 n136-128-154:801240:802613 [0] NCCL INFO 10 : GPU/0-89000 GPU/0-8e000 n136-128-154:801240:802613 [0] NCCL INFO 11 : GPU/0-89000 GPU/0-8e000 n136-128-154:801240:802613 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0 n136-128-154:801240:802613 [0] NCCL INFO 0 : GPU/0-89000 GPU/0-8e000 n136-128-154:801240:802613 [0] NCCL INFO 1 : GPU/0-89000 GPU/0-8e000 n136-128-154:801240:802613 [0] NCCL INFO 2 : GPU/0-89000 GPU/0-8e000 n136-128-154:801240:802613 [0] NCCL INFO 3 : GPU/0-89000 GPU/0-8e000 n136-128-154:801240:802613 [0] NCCL INFO 4 : GPU/0-89000 GPU/0-8e000 n136-128-154:801240:802613 [0] NCCL INFO 5 : GPU/0-89000 GPU/0-8e000 n136-128-154:801240:802613 [0] NCCL INFO 6 : GPU/0-8e000 GPU/0-89000 n136-128-154:801240:802613 [0] NCCL INFO 7 : GPU/0-8e000 GPU/0-89000 n136-128-154:801240:802613 [0] NCCL INFO 8 : GPU/0-8e000 GPU/0-89000 n136-128-154:801240:802613 [0] NCCL INFO 9 : GPU/0-8e000 GPU/0-89000 n136-128-154:801240:802613 [0] NCCL INFO 10 : GPU/0-8e000 GPU/0-89000 n136-128-154:801240:802613 [0] NCCL INFO 11 : GPU/0-8e000 GPU/0-89000 n136-128-154:801241:802665 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 20.000000/20.000000, type NVL/PIX, sameChannels 1 n136-128-154:801241:802665 [1] NCCL INFO 0 : GPU/0-89000 GPU/0-8e000 n136-128-154:801241:802665 [1] NCCL INFO 1 : GPU/0-89000 GPU/0-8e000 n136-128-154:801241:802665 [1] NCCL INFO 2 : GPU/0-89000 GPU/0-8e000 n136-128-154:801241:802665 [1] NCCL INFO 3 : GPU/0-89000 GPU/0-8e000 n136-128-154:801241:802665 [1] NCCL INFO 4 : GPU/0-89000 GPU/0-8e000 n136-128-154:801241:802665 [1] NCCL INFO 5 : GPU/0-89000 GPU/0-8e000 n136-128-154:801241:802665 [1] NCCL INFO 6 : GPU/0-89000 GPU/0-8e000 n136-128-154:801241:802665 [1] NCCL INFO 7 : GPU/0-89000 GPU/0-8e000 n136-128-154:801241:802665 [1] NCCL INFO 8 : GPU/0-89000 GPU/0-8e000 n136-128-154:801241:802665 [1] NCCL INFO 9 : GPU/0-89000 GPU/0-8e000 n136-128-154:801241:802665 [1] NCCL INFO 10 : GPU/0-89000 GPU/0-8e000 n136-128-154:801241:802665 [1] NCCL INFO 11 : GPU/0-89000 GPU/0-8e000 n136-128-154:801241:802665 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 40.000000/40.000000, type NVL/PIX, sameChannels 0 n136-128-154:801241:802665 [1] NCCL INFO 0 : GPU/0-89000 GPU/0-8e000 n136-128-154:801241:802665 [1] NCCL INFO 1 : GPU/0-89000 GPU/0-8e000 n136-128-154:801241:802665 [1] NCCL INFO 2 : GPU/0-89000 GPU/0-8e000 n136-128-154:801241:802665 [1] NCCL INFO 3 : GPU/0-89000 GPU/0-8e000 n136-128-154:801241:802665 [1] NCCL INFO 4 : GPU/0-89000 GPU/0-8e000 n136-128-154:801241:802665 [1] NCCL INFO 5 : GPU/0-89000 GPU/0-8e000 n136-128-154:801241:802665 [1] NCCL INFO 6 : GPU/0-8e000 GPU/0-89000 n136-128-154:801241:802665 [1] NCCL INFO 7 : GPU/0-8e000 GPU/0-89000 n136-128-154:801241:802665 [1] NCCL INFO 8 : GPU/0-8e000 GPU/0-89000 n136-128-154:801241:802665 [1] NCCL INFO 9 : GPU/0-8e000 GPU/0-89000 n136-128-154:801241:802665 [1] NCCL INFO 10 : GPU/0-8e000 GPU/0-89000 n136-128-154:801241:802665 [1] NCCL INFO 11 : GPU/0-8e000 GPU/0-89000 n136-128-154:801240:802613 [0] NCCL INFO comm 0x3e4064c0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0 n136-128-154:801241:802665 [1] NCCL INFO comm 0x11f84b90 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0 n136-128-154:801240:802613 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1 n136-128-154:801240:802613 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1 n136-128-154:801241:802665 [1] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1 n136-128-154:801240:802613 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1 n136-128-154:801241:802665 [1] NCCL INFO Tree 12 : 0 -> 1 -> -1/-1/-1 n136-128-154:801240:802613 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1 n136-128-154:801241:802665 [1] NCCL INFO Tree 1 : 0 -> 1 -> -1/-1/-1 n136-128-154:801240:802613 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1 n136-128-154:801241:802665 [1] NCCL INFO Tree 13 : 0 -> 1 -> -1/-1/-1 n136-128-154:801240:802613 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1 n136-128-154:801241:802665 [1] NCCL INFO Tree 2 : 0 -> 1 -> -1/-1/-1 n136-128-154:801240:802613 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1 n136-128-154:801241:802665 [1] NCCL INFO Tree 14 : 0 -> 1 -> -1/-1/-1 n136-128-154:801240:802613 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1 n136-128-154:801241:802665 [1] NCCL INFO Tree 3 : 0 -> 1 -> -1/-1/-1 n136-128-154:801240:802613 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1 n136-128-154:801241:802665 [1] NCCL INFO Tree 15 : 0 -> 1 -> -1/-1/-1 n136-128-154:801240:802613 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1 n136-128-154:801241:802665 [1] NCCL INFO Tree 4 : 0 -> 1 -> -1/-1/-1 n136-128-154:801240:802613 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1 n136-128-154:801241:802665 [1] NCCL INFO Tree 16 : 0 -> 1 -> -1/-1/-1 n136-128-154:801240:802613 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1 n136-128-154:801241:802665 [1] NCCL INFO Tree 5 : 0 -> 1 -> -1/-1/-1 n136-128-154:801240:802613 [0] NCCL INFO Tree 6 : 1 -> 0 -> -1/-1/-1 n136-128-154:801241:802665 [1] NCCL INFO Tree 17 : 0 -> 1 -> -1/-1/-1 n136-128-154:801240:802613 [0] NCCL INFO Tree 18 : 1 -> 0 -> -1/-1/-1 n136-128-154:801241:802665 [1] NCCL INFO Tree 6 : -1 -> 1 -> 0/-1/-1 n136-128-154:801240:802613 [0] NCCL INFO Tree 7 : 1 -> 0 -> -1/-1/-1 n136-128-154:801241:802665 [1] NCCL INFO Tree 18 : -1 -> 1 -> 0/-1/-1 n136-128-154:801240:802613 [0] NCCL INFO Tree 19 : 1 -> 0 -> -1/-1/-1 n136-128-154:801241:802665 [1] NCCL INFO Tree 7 : -1 -> 1 -> 0/-1/-1 n136-128-154:801240:802613 [0] NCCL INFO Tree 8 : 1 -> 0 -> -1/-1/-1 n136-128-154:801241:802665 [1] NCCL INFO Tree 19 : -1 -> 1 -> 0/-1/-1 n136-128-154:801240:802613 [0] NCCL INFO Tree 20 : 1 -> 0 -> -1/-1/-1 n136-128-154:801241:802665 [1] NCCL INFO Tree 8 : -1 -> 1 -> 0/-1/-1 n136-128-154:801240:802613 [0] NCCL INFO Tree 9 : 1 -> 0 -> -1/-1/-1 n136-128-154:801241:802665 [1] NCCL INFO Tree 20 : -1 -> 1 -> 0/-1/-1 n136-128-154:801240:802613 [0] NCCL INFO Tree 21 : 1 -> 0 -> -1/-1/-1 n136-128-154:801241:802665 [1] NCCL INFO Tree 9 : -1 -> 1 -> 0/-1/-1 n136-128-154:801240:802613 [0] NCCL INFO Tree 10 : 1 -> 0 -> -1/-1/-1 n136-128-154:801241:802665 [1] NCCL INFO Tree 21 : -1 -> 1 -> 0/-1/-1 n136-128-154:801240:802613 [0] NCCL INFO Tree 22 : 1 -> 0 -> -1/-1/-1 n136-128-154:801241:802665 [1] NCCL INFO Tree 10 : -1 -> 1 -> 0/-1/-1 n136-128-154:801240:802613 [0] NCCL INFO Tree 11 : 1 -> 0 -> -1/-1/-1 n136-128-154:801241:802665 [1] NCCL INFO Tree 22 : -1 -> 1 -> 0/-1/-1 n136-128-154:801240:802613 [0] NCCL INFO Tree 23 : 1 -> 0 -> -1/-1/-1 n136-128-154:801241:802665 [1] NCCL INFO Tree 11 : -1 -> 1 -> 0/-1/-1 n136-128-154:801241:802665 [1] NCCL INFO Tree 23 : -1 -> 1 -> 0/-1/-1 n136-128-154:801240:802613 [0] NCCL INFO Channel 00/24 : 0 1 n136-128-154:801240:802613 [0] NCCL INFO Channel 01/24 : 0 1 n136-128-154:801240:802613 [0] NCCL INFO Channel 02/24 : 0 1 n136-128-154:801241:802665 [1] NCCL INFO Ring 00 : 0 -> 1 -> 0 n136-128-154:801240:802613 [0] NCCL INFO Channel 03/24 : 0 1 n136-128-154:801241:802665 [1] NCCL INFO Ring 01 : 0 -> 1 -> 0 n136-128-154:801240:802613 [0] NCCL INFO Channel 04/24 : 0 1 n136-128-154:801241:802665 [1] NCCL INFO Ring 02 : 0 -> 1 -> 0 n136-128-154:801240:802613 [0] NCCL INFO Channel 05/24 : 0 1 n136-128-154:801241:802665 [1] NCCL INFO Ring 03 : 0 -> 1 -> 0 n136-128-154:801240:802613 [0] NCCL INFO Channel 06/24 : 0 1 n136-128-154:801241:802665 [1] NCCL INFO Ring 04 : 0 -> 1 -> 0 n136-128-154:801240:802613 [0] NCCL INFO Channel 07/24 : 0 1 n136-128-154:801241:802665 [1] NCCL INFO Ring 05 : 0 -> 1 -> 0 n136-128-154:801240:802613 [0] NCCL INFO Channel 08/24 : 0 1 n136-128-154:801241:802665 [1] NCCL INFO Ring 06 : 0 -> 1 -> 0 n136-128-154:801240:802613 [0] NCCL INFO Channel 09/24 : 0 1 n136-128-154:801241:802665 [1] NCCL INFO Ring 07 : 0 -> 1 -> 0 n136-128-154:801240:802613 [0] NCCL INFO Channel 10/24 : 0 1 n136-128-154:801241:802665 [1] NCCL INFO Ring 08 : 0 -> 1 -> 0 n136-128-154:801240:802613 [0] NCCL INFO Channel 11/24 : 0 1 n136-128-154:801241:802665 [1] NCCL INFO Ring 09 : 0 -> 1 -> 0 n136-128-154:801240:802613 [0] NCCL INFO Channel 12/24 : 0 1 n136-128-154:801241:802665 [1] NCCL INFO Ring 10 : 0 -> 1 -> 0 n136-128-154:801240:802613 [0] NCCL INFO Channel 13/24 : 0 1 n136-128-154:801241:802665 [1] NCCL INFO Ring 11 : 0 -> 1 -> 0 n136-128-154:801240:802613 [0] NCCL INFO Channel 14/24 : 0 1 n136-128-154:801241:802665 [1] NCCL INFO Ring 12 : 0 -> 1 -> 0 n136-128-154:801240:802613 [0] NCCL INFO Channel 15/24 : 0 1 n136-128-154:801241:802665 [1] NCCL INFO Ring 13 : 0 -> 1 -> 0 n136-128-154:801240:802613 [0] NCCL INFO Channel 16/24 : 0 1 n136-128-154:801241:802665 [1] NCCL INFO Ring 14 : 0 -> 1 -> 0 n136-128-154:801240:802613 [0] NCCL INFO Channel 17/24 : 0 1 n136-128-154:801241:802665 [1] NCCL INFO Ring 15 : 0 -> 1 -> 0 n136-128-154:801240:802613 [0] NCCL INFO Channel 18/24 : 0 1 n136-128-154:801241:802665 [1] NCCL INFO Ring 16 : 0 -> 1 -> 0 n136-128-154:801240:802613 [0] NCCL INFO Channel 19/24 : 0 1 n136-128-154:801241:802665 [1] NCCL INFO Ring 17 : 0 -> 1 -> 0 n136-128-154:801240:802613 [0] NCCL INFO Channel 20/24 : 0 1 n136-128-154:801241:802665 [1] NCCL INFO Ring 18 : 0 -> 1 -> 0 n136-128-154:801240:802613 [0] NCCL INFO Channel 21/24 : 0 1 n136-128-154:801241:802665 [1] NCCL INFO Ring 19 : 0 -> 1 -> 0 n136-128-154:801240:802613 [0] NCCL INFO Channel 22/24 : 0 1 n136-128-154:801241:802665 [1] NCCL INFO Ring 20 : 0 -> 1 -> 0 n136-128-154:801240:802613 [0] NCCL INFO Channel 23/24 : 0 1 n136-128-154:801241:802665 [1] NCCL INFO Ring 21 : 0 -> 1 -> 0 n136-128-154:801240:802613 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1 n136-128-154:801241:802665 [1] NCCL INFO Ring 22 : 0 -> 1 -> 0 n136-128-154:801240:802613 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1 n136-128-154:801241:802665 [1] NCCL INFO Ring 23 : 0 -> 1 -> 0 n136-128-154:801240:802613 [0] NCCL INFO Ring 02 : 1 -> 0 -> 1 n136-128-154:801241:802665 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1 n136-128-154:801240:802613 [0] NCCL INFO Ring 03 : 1 -> 0 -> 1 n136-128-154:801240:802613 [0] NCCL INFO Ring 04 : 1 -> 0 -> 1 n136-128-154:801241:802665 [1] NCCL INFO P2P Chunksize set to 524288 n136-128-154:801240:802613 [0] NCCL INFO Ring 05 : 1 -> 0 -> 1 n136-128-154:801240:802613 [0] NCCL INFO Ring 06 : 1 -> 0 -> 1 n136-128-154:801240:802613 [0] NCCL INFO Ring 07 : 1 -> 0 -> 1 n136-128-154:801240:802613 [0] NCCL INFO Ring 08 : 1 -> 0 -> 1 n136-128-154:801240:802613 [0] NCCL INFO Ring 09 : 1 -> 0 -> 1 n136-128-154:801240:802613 [0] NCCL INFO Ring 10 : 1 -> 0 -> 1 n136-128-154:801240:802613 [0] NCCL INFO Ring 11 : 1 -> 0 -> 1 n136-128-154:801240:802613 [0] NCCL INFO Ring 12 : 1 -> 0 -> 1 n136-128-154:801240:802613 [0] NCCL INFO Ring 13 : 1 -> 0 -> 1 n136-128-154:801240:802613 [0] NCCL INFO Ring 14 : 1 -> 0 -> 1 n136-128-154:801240:802613 [0] NCCL INFO Ring 15 : 1 -> 0 -> 1 n136-128-154:801240:802613 [0] NCCL INFO Ring 16 : 1 -> 0 -> 1 n136-128-154:801240:802613 [0] NCCL INFO Ring 17 : 1 -> 0 -> 1 n136-128-154:801240:802613 [0] NCCL INFO Ring 18 : 1 -> 0 -> 1 n136-128-154:801240:802613 [0] NCCL INFO Ring 19 : 1 -> 0 -> 1 n136-128-154:801240:802613 [0] NCCL INFO Ring 20 : 1 -> 0 -> 1 n136-128-154:801240:802613 [0] NCCL INFO Ring 21 : 1 -> 0 -> 1 n136-128-154:801240:802613 [0] NCCL INFO Ring 22 : 1 -> 0 -> 1 n136-128-154:801240:802613 [0] NCCL INFO Ring 23 : 1 -> 0 -> 1 n136-128-154:801240:802613 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1 n136-128-154:801240:802613 [0] NCCL INFO P2P Chunksize set to 524288 n136-128-154:801240:802613 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. n136-128-154:801240:802613 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 n136-128-154:801240:802693 [0] NCCL INFO [Proxy Service] Device 0 CPU core 39 n136-128-154:801240:802694 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 104 n136-128-154:801241:802665 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. n136-128-154:801241:802696 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 99 n136-128-154:801241:802695 [1] NCCL INFO [Proxy Service] Device 1 CPU core 98 n136-128-154:801240:802613 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 n136-128-154:801240:802613 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer n136-128-154:801240:802613 [0] NCCL INFO CC Off, workFifoBytes 1048576 n136-128-154:801241:802665 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 n136-128-154:801241:802665 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer n136-128-154:801240:802613 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. n136-128-154:801240:802613 [0] NCCL INFO ncclCommInitRankConfig comm 0x3e4064c0 rank 0 nranks 2 cudaDev 0 nvmlDev 4 busId 89000 commId 0xd6174eb4c702f7 - Init COMPLETE n136-128-154:801240:802613 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 4.29 (kernels 0.14, alloc 0.19, bootstrap 3.74, allgathers 0.01, topo 0.08, graphs 0.00, connections 0.01, rest 0.12) n136-128-154:801241:802665 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. n136-128-154:801241:802665 [1] NCCL INFO ncclCommInitRankConfig comm 0x11f84b90 rank 1 nranks 2 cudaDev 1 nvmlDev 5 busId 8e000 commId 0xd6174eb4c702f7 - Init COMPLETE n136-128-154:801241:802665 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.62 (kernels 0.19, alloc 0.22, bootstrap 0.00, allgathers 0.00, topo 0.08, graphs 0.00, connections 0.01, rest 0.12) n136-128-154:801240:802697 [0] NCCL INFO Channel 00/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:801240:802697 [0] NCCL INFO Channel 01/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:801240:802697 [0] NCCL INFO Channel 02/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:801240:802697 [0] NCCL INFO Channel 03/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:801240:802697 [0] NCCL INFO Channel 04/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:801240:802697 [0] NCCL INFO Channel 05/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:801240:802697 [0] NCCL INFO Channel 06/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:801240:802697 [0] NCCL INFO Channel 07/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Channel 00/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801240:802697 [0] NCCL INFO Channel 08/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Channel 01/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801240:802697 [0] NCCL INFO Channel 09/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Channel 02/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801240:802697 [0] NCCL INFO Channel 10/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Channel 03/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801240:802697 [0] NCCL INFO Channel 11/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Channel 04/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801240:802697 [0] NCCL INFO Channel 12/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Channel 05/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801240:802697 [0] NCCL INFO Channel 13/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Channel 06/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801240:802697 [0] NCCL INFO Channel 14/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Channel 07/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801240:802697 [0] NCCL INFO Channel 15/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Channel 08/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801240:802697 [0] NCCL INFO Channel 16/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Channel 09/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801240:802697 [0] NCCL INFO Channel 17/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Channel 10/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801240:802697 [0] NCCL INFO Channel 18/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Channel 11/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801240:802697 [0] NCCL INFO Channel 19/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Channel 12/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801240:802697 [0] NCCL INFO Channel 20/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Channel 13/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801240:802697 [0] NCCL INFO Channel 21/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Channel 14/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801240:802697 [0] NCCL INFO Channel 22/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Channel 15/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801240:802697 [0] NCCL INFO Channel 23/0 : 0[4] -> 1[5] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Channel 16/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Channel 17/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Channel 18/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Channel 19/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Channel 20/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Channel 21/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Channel 22/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Channel 23/0 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:802698 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 n136-128-154:801240:802697 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 2025-12-01:21:10:13 INFO [evaluator:559] Running loglikelihood requests 2025-12-01:21:10:13 INFO [evaluator:559] Running loglikelihood requests Running loglikelihood requests: 0%| | 0/20084 [00:00 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 01/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 02/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 03/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 04/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 05/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 06/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 07/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 08/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 09/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 10/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 11/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 12/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 13/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 14/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 15/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 16/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 17/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 18/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 19/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 20/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 21/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 22/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 23/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 24/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 25/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 26/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 27/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 28/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 29/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 30/1 : 1[5] -> 0[4] via P2P/CUMEM/read n136-128-154:801241:816089 [1] NCCL INFO Channel 31/1 : 1[5] -> 0[4] via P2P/CUMEM/read fatal: detected dubious ownership in repository at '/mnt/bn/life-mllm/users/cxr/quantization' To add an exception for this directory, call: git config --global --add safe.directory /mnt/bn/life-mllm/users/cxr/quantization n136-128-154:801241:816094 [1] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:801241:816094 [1] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:801241:816094 [1] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:801241:816094 [1] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:801241:816094 [1] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:801241:816094 [1] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:801241:802695 [1] NCCL INFO misc/socket.cc:915 -> 3 2025-12-01:21:54:52 INFO [loggers.evaluation_tracker:209] Saving results aggregated hf (pretrained=/mnt/bn/life-mllm/users/cxr/quantization/models/Qwen/Qwen2.5-14B), gen_kwargs: (None), limit: None, num_fewshot: 10, batch_size: auto (32) | Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr| |---------|------:|------|-----:|--------|---|-----:|---|-----:| |hellaswag| 1|none | 10|acc |↑ |0.6432|± |0.0048| | | |none | 10|acc_norm|↑ |0.8425|± |0.0036| [rank0]:[W1201 21:54:53.255015824 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) n136-128-154:801240:816144 [0] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:801240:816144 [0] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:801240:816144 [0] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:801240:802693 [0] NCCL INFO misc/socket.cc:915 -> 3 n136-128-154:801240:816144 [0] NCCL INFO misc/socket.cc:64 -> 3 n136-128-154:801240:816144 [0] NCCL INFO misc/socket.cc:81 -> 3 n136-128-154:801240:816144 [0] NCCL INFO misc/socket.cc:863 -> 3 n136-128-154:801241:802695 [1] NCCL INFO misc/socket.cc:915 -> 3 n136-128-154:801241:816094 [1] NCCL INFO comm 0x11f84b90 rank 1 nranks 2 cudaDev 1 busId 8e000 - Abort COMPLETE n136-128-154:801240:816144 [0] NCCL INFO comm 0x3e4064c0 rank 0 nranks 2 cudaDev 0 busId 89000 - Abort COMPLETE 任务 hellaswag 评估完成!