[DCP] Improve with_temp_dir robustness (#111106)
Calling os.sync() to ensure the tempfile can be seens across ranks.
Differential Revision: [D50209697](https://our.internmc.facebook.com/intern/diff/D50209697/)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/111106
Approved by: https://github.com/Skylion007, https://github.com/wz337
