pytorch

65314a61 - [c10d] add an unit test for unordered destruction of PGs (#119045)

[c10d] add an unit test for unordered destruction of PGs (#119045) Summary: We were suspecting ncclCommsAbort was hung due to NCCL 2.17's 'bug' triggered by different ranks calls desctructors of different PGs in different order. This can be reproed in a NCCL level test for 2.17 We need a test case in c10d to constantly check if PGs can be destructed in different order Test Plan: Run the test and print out the distruction orders are expected ``` [$ python test/distributed/test_c10d_nccl.py ProcessGroupNCCLTest.test_close_multi_pg_unordered NCCL version 2.19.3+cuda12.0 [rank0]:[W ProcessGroupNCCL.cpp:1128] [PG 2 Rank 0] ProcessGroupNCCL destructor entered. [rank0]:[W ProcessGroupNCCL.cpp:1147] [PG 2 Rank 0] ProcessGroupNCCL aborting communicators, check for 'abort finished' logs or look for abort hang [rank1]:[W ProcessGroupNCCL.cpp:1128] [PG 1 Rank 1] ProcessGroupNCCL destructor entered. [rank1]:[W ProcessGroupNCCL.cpp:1147] [PG 1 Rank 1] ProcessGroupNCCL aborting communicators, check for 'abort finished' logs or look for abort hang [rank0]:[W ProcessGroupNCCL.cpp:1151] [PG 2 Rank 0] ProcessGroupNCCL abort finished. [rank0]:[W ProcessGroupNCCL.cpp:1128] [PG 1 Rank 0] ProcessGroupNCCL destructor entered. [rank0]:[W ProcessGroupNCCL.cpp:1147] [PG 1 Rank 0] ProcessGroupNCCL aborting communicators, check for 'abort finished' logs or look for abort hang [rank1]:[W ProcessGroupNCCL.cpp:1151] [PG 1 Rank 1] ProcessGroupNCCL abort finished. [rank1]:[W ProcessGroupNCCL.cpp:1128] [PG 2 Rank 1] ProcessGroupNCCL destructor entered. [rank1]:[W ProcessGroupNCCL.cpp:1147] [PG 2 Rank 1] ProcessGroupNCCL aborting communicators, check for 'abort finished' logs or look for abort hang [rank0]:[W ProcessGroupNCCL.cpp:1151] [PG 1 Rank 0] ProcessGroupNCCL abort finished. [rank1]:[W ProcessGroupNCCL.cpp:1151] [PG 2 Rank 1] ProcessGroupNCCL abort finished. . ---------------------------------------------------------------------- Ran 1 test in 18.969s OK](url) ``` Pull Request resolved: https://github.com/pytorch/pytorch/pull/119045 Approved by: https://github.com/yifuwang