[redo] Fix SyncBatchNorm forward pass for non-default process group (#43861)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/43861
This is a redo of https://github.com/pytorch/pytorch/pull/38874, and
fixing my original bug from
https://github.com/pytorch/pytorch/pull/38246.
Test Plan:
CI
Imported from OSS
Reviewed By: supriyar
Differential Revision: D23418816
fbshipit-source-id: 2a3a3d67fc2d03bb0bf30a87cce4e805ac8839fb
