Revert "Fix test failure in TestCudaMultiGPU.test_cuda_device_memory_allocated (#105501)"
This reverts commit e6fd8ca3eef2b85b821936829e86beb7d832575c.
Reverted https://github.com/pytorch/pytorch/pull/105501 on behalf of https://github.com/zou3519 due to We've agreed that the PR is wrong. It didn't actually break anything. ([comment](https://github.com/pytorch/pytorch/pull/105501#issuecomment-1648005842))
