[DDP] fix test_ddp_inference (#61666)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/61666
Closes https://github.com/pytorch/pytorch/issues/61481. Fixes this
test by removing section that uses only torch.no_grad() and doesn't call
model.eval(). For SyncBN, need to call model.eval() otherwise SyncBN will
assume it is in training mode, which does collective calls in the forward pass
and does not work for inference.
ghstack-source-id: 133657549
Test Plan: CI
Reviewed By: mrshenli
Differential Revision: D29699444
fbshipit-source-id: 03ccb296dd9cb56729cd23e91c7f50b72fcf3adf
