pytorch
906a1952 - [DDP] Enable delayed all reduce in DDP (#96673)

Commit
2 years ago
[DDP] Enable delayed all reduce in DDP (#96673) Summary: Enable the functionality of delaying all reduce in DDP to specify the parameters whose all reduce will be hooked to a specific param. This prevents AllReduce blocking All2All in some recommendation models. Test Plan: GitHub CI. Pull Request resolved: https://github.com/pytorch/pytorch/pull/96673 Approved by: https://github.com/zhaojuanmao
Author
Committer
Parents
Loading