Make WorkNCCL use CUDAEvent::query() rather than re-implement it (#49343)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/49343
at::cuda::CUDAEvent is "lazy" and only creates an event when it's first recorded. Until then, at::cuda::CUDAEvent is empty. If we use at::cuda::CUDAEvent::query() this is taken into account (an empty event is always ready), but WorkNCCL extracts the raw cudaEvent_t value from at::cuda::CUDAEvent and calls cudaEventQuery manually and doesn't check this. This could cause a failure.
It's unclear if this is ever supposed to happen, but we're seeing that failure, and we want to sort it out in order to see if there's something "deeper" going on.
ghstack-source-id: 118532806
Test Plan: Unit tests
Reviewed By: SciPioneer
Differential Revision: D25537844
fbshipit-source-id: 506319f4742e1c0a02aa75ecc01112ea3be42d8f
