[Prims] Unbreak CUDA lazy init (#80899)
CUDA calls should not be made in the default codepath
Fixes https://github.com/pytorch/pytorch/issues/80876
Pull Request resolved: https://github.com/pytorch/pytorch/pull/80899
Approved by: https://github.com/ngimel
