Add All-to-all comms support to distributed module and MPI backend (#32361)
Summary:
As described in https://github.com/pytorch/pytorch/issues/32345, a prototype implementation to add an alltoall communication primitive to torch.distributed module and ProcessGroup abstract interface. Also, implements alltoall in ProcessGroupMPI backend.
mnaumovfb JianpingChen066 dmudiger srinivas212 Jianhui-Li mshiryaev ftian1
cc pietern mrshenli pritamdamania87 zhaojuanmao satgera rohan-varma gqchen aazzolini xush6528 osalpekar
Pull Request resolved: https://github.com/pytorch/pytorch/pull/32361
Reviewed By: mrshenli
Differential Revision: D20635481
Pulled By: srinivas212
fbshipit-source-id: 3dd0af800ce55d02f02813cde550e3a0f1a287d2
