Add architectural support for multi-GPU. (#44059)
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/44059
Test Plan: Imported from OSS
Reviewed By: IvanKobzarev
Differential Revision: D23820825
Pulled By: AshkanAliabadi
fbshipit-source-id: 0719b00581487a77ebadff867d1e4ac89354bf90
