Call uncheckedSetDevice in ~InlineDeviceGuard only when device index are different (#35438)
Summary:
Setting device could be expensive, especially when a debugger is present. We should check the device are different before we set.
cc: ptrblck
Pull Request resolved: https://github.com/pytorch/pytorch/pull/35438
Differential Revision: D20664084
Pulled By: ngimel
fbshipit-source-id: 2440b4c9d96c41b4a19d5b1e8e1756fa40f090f0
