pytorch

752ab799 - Support noncontiguous inputs for torch.distributed.nn.functional.all_gather/reducescatter/gather

Support noncontiguous inputs for torch.distributed.nn.functional.all_gather/reducescatter/gather Fixes #73515 The backward for AllGather is ReduceScatter. I am wondering is there a deeper reason why it's currently implemented as All2All with explicit sum. ReduceScatter also has a lower communication payload than All2All. In addition, dist.reduce_scatter accepts non-contiguous input_tensor_list. Pull Request resolved: https://github.com/pytorch/pytorch/pull/75276 Approved by: https://github.com/H-Huang