transformers
d8894645 - Fix zero expert gradients in bfloat16 with expert parallelism and ep_size < fsdp_size (#49688)

Commit
28 days ago
Fix zero expert gradients in bfloat16 with expert parallelism and ep_size < fsdp_size (#49688) * Fix zero expert gradients in bfloat16 with expert parallelism and ep_size < fsdp_size * Rewrite the expert gradient reduction comments * Restore the expert gradient divide factor comment * Compare the loss and every gradient in the bf16 expert parallel test --------- Co-authored-by: Ferdinand Mom <47445085+3outeille@users.noreply.github.com>
Author
Parents
Loading