[BE] Fix flaky test_ddp_model_diff_across_ranks test (#61546)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/61546
Closes https://github.com/pytorch/pytorch/issues/60661
Fixes this flaky test by using blocking wait instead of async error handling, and performs a gloo-based barrier with higher timeout at the end of test which avoids issues with Barrier.sync. This also allows us to remove this test from the `skip_return_code_checks` list.
ghstack-source-id: 133657107
Test Plan: CI
Reviewed By: zhaojuanmao
Differential Revision: D29663884
fbshipit-source-id: 9f0df085b1968f6a7e2c7ae2f06b6dcd4838a87e
