Hello, as the title says, I'm experiencing an NCC timeout issue when training on 4/1 machine with 8 GPUs using open source code. Has anyone else encountered this issue?
I can almost reliably reproduce the timeout issue at a fixed number of steps. The error message is as follows:
[Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1892, OpType=ALLREDUCE, NumelIn=574105888, NumelOut=574105888, Timeout(ms)=600000) ran for 600016 milliseconds before timing out.
【2025-09-25 14:27:49】[rank2]:[E ProcessGroupNCCL.cpp:563] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1892, OpType=ALLREDUCE, NumelIn=574105888, NumelOut=574105888, Timeout(ms)=600000) ran for 600034 milliseconds before timing out.
【2025-09-25 14:27:49】[rank3]:[E ProcessGroupNCCL.cpp:563] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1892, OpType=ALLREDUCE, NumelIn=574105888, NumelOut=574105888, Timeout(ms)=600000) ran for 600025 milliseconds before timing out.
【2025-09-25 14:27:49】[rank5]:[E ProcessGroupNCCL.cpp:563] [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1892, OpType=ALLREDUCE, NumelIn=574105888, NumelOut=574105888, Timeout(ms)=600000) ran for 600045 milliseconds before timing out.
【2025-09-25 14:27:49】[rank4]:[E ProcessGroupNCCL.cpp:563] [Rank 4] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1892, OpType=ALLREDUCE, NumelIn=574105888, NumelOut=574105888, Timeout(ms)=600000) ran for 600024 milliseconds before timing out.
【2025-09-25 14:27:49】[rank7]:[E ProcessGroupNCCL.cpp:563] [Rank 7] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1892, OpType=ALLREDUCE, NumelIn=574105888, NumelOut=574105888, Timeout(ms)=600000) ran for 600050 milliseconds before timing out.
【2025-09-25 14:27:49】[rank1]:[E ProcessGroupNCCL.cpp:563] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1892, OpType=ALLREDUCE, NumelIn=574105888, NumelOut=574105888, Timeout(ms)=600000) ran for 600067 milliseconds before timing out.
【2025-09-25 14:27:49】[rank6]:[E ProcessGroupNCCL.cpp:563] [Rank 6] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1892, OpType=ALLREDUCE, NumelIn=574105888, NumelOut=574105888, Timeout(ms)=600000) ran for 600080 milliseconds before timing out.
【2025-09-25 14:27:50】[rank3]:[E ProcessGroupNCCL.cpp:1537] [PG 1 Rank 3] Timeout at NCCL work: 1892, last enqueued NCCL work: 1894, last completed NCCL work: 1891.
【2025-09-25 14:27:50】[rank5]:[E ProcessGroupNCCL.cpp:1537] [PG 1 Rank 5] Timeout at NCCL work: 1892, last enqueued NCCL work: 1894, last completed NCCL work: 1891.
【2025-09-25 14:27:50】[rank5]:[E ProcessGroupNCCL.cpp:577] [Rank 5] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
【2025-09-25 14:27:50】[rank3]:[E ProcessGroupNCCL.cpp:577] [Rank 3] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
【2025-09-25 14:27:50】[rank3]:[E ProcessGroupNCCL.cpp:583] [Rank 3] To avoid data inconsistency, we are taking the entire process down.
【2025-09-25 14:27:50】[rank5]:[E ProcessGroupNCCL.cpp:583] [Rank 5] To avoid data inconsistency, we are taking the entire process down.
【2025-09-25 14:27:50】[rank3]:[E ProcessGroupNCCL.cpp:1414] [PG 1 Rank 3] Process group watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1892, OpType=ALLREDUCE, NumelIn=574105888, NumelOut=574105888, Timeout(ms)=600000) ran for 600025 milliseconds before timing out.
【2025-09-25 14:27:50】Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first):
【2025-09-25 14:27:50】frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f990357a897 in software/anaconda3/envs/vita/lib/python3.10/site-packages/torch/lib/libc10.so)
【2025-09-25 14:27:50】frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f98b7077c62 in software/anaconda3/envs/vita/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
【2025-09-25 14:27:50】frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7f98b707ca80 in software/anaconda3/envs/vita/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
【2025-09-25 14:27:50】frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f98b707ddcc in software/anaconda3/envs/vita/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
【2025-09-25 14:27:50】frame #4: <unknown function> + 0xdc253 (0x7f9912a61253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
【2025-09-25 14:27:50】frame #5: <unknown function> + 0x8a8b2 (0x7f9934f268b2 in /usr/lib/x86_64-linux-gnu/libc.so.6)
【2025-09-25 14:27:50】frame #6: clone + 0x44 (0x7f9934fab184 in /usr/lib/x86_64-linux-gnu/libc.so.6)
Hello, as the title says, I'm experiencing an NCC timeout issue when training on 4/1 machine with 8 GPUs using open source code. Has anyone else encountered this issue?
I can almost reliably reproduce the timeout issue at a fixed number of steps. The error message is as follows: