DeepSpeed
[AutoEP] Run staged expert weight gradients with ZeRO-3
#9115
Open

Commits
  • [AutoEP] Run staged expert weight gradients with ZeRO-3
    PKUWZP committed 14 hours ago
  • [AutoEP] Do not keep gathered expert weights allocated until the backward
    PKUWZP committed 14 hours ago
Loading