[NCCL] enable p2p tests (#47797)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/47797
NCCL p2p tests had hang issues before, the reason is that there were some unexpected context switches. For example, process 1 which is supposed to only use GPU1 could use GPU0 as a result of missing explicitly setting device.
ghstack-source-id: 116461969
Test Plan: waitforsandcastle
Reviewed By: jiayisuse
Differential Revision: D24863808
fbshipit-source-id: 92bd3a4874be8334210c7c8ee6363648893c963e
