Skip accelerator sync for unprofiled comm ops (#8795)
Fixes #8756
`timed_op` called `get_accelerator().synchronize()` for every wrapped
comm op whenever comms logging was enabled. Ops not selected by
`prof_ops`, `prof_all` or `prof` get no timer and no record, so the sync
was pure overhead.
The sync is now guarded by `should_profile`. The MPI `cdb.barrier()` is
unchanged.
Test: `test_collective_profiling` now counts `synchronize()` calls. It
must be 0 for the `unselected` case and 1 for `prof_all`, `prof`, and
both `prof_ops` cases. Without the fix the 8 `unselected` cases fail;
with it all 52 tests in `test_comms_logger.py` pass. yapf is clean.
Verified on a node with A40 GPUs, but the tests use a mocked backend and
accelerator, so no GPU was exercised. Not verified: real NCCL/MPI runs,
training step timing, or the rest of the unit test suite.
Thanks to @marcusacosta for the report.
This change was prepared by an automated AI agent (Claude) and verified
as described above.
Signed-off-by: guptaishaan <guptaishaan@users.noreply.github.com>
Co-authored-by: guptaishaan <guptaishaan@users.noreply.github.com>
