transformers
201d5624 - Compute MoE load-balancing loss per layer to avoid giant one-hot materialization

Commit

3 days ago

Compute MoE load-balancing loss per layer to avoid giant one-hot materialization

References

fix-moe-aux-loss-memory

#48131 - Compute MoE load-balancing loss per layer to avoid giant one-hot materialization −99.7% @ 128k

Author

qgallouedec

qgallouedec

Committer

qgallouedec

qgallouedec

Parents

Loading