DeepSpeed
27228462 - [BUGFIX] Reset `bucket.elements` after reduction in ZeRO Stage 3 (#7418)

Commit
176 days ago
[BUGFIX] Reset `bucket.elements` after reduction in ZeRO Stage 3 (#7418) closes #7415 # Description Resets `bucket.elements` after reduction in ZeRO Stage 3. Without this, the bucket grows indefinitely, reducing only one param at a time. Added `bucket.elements = 0` after `params_in_bucket.clear()`. Co-authored-by: a <a>
Author
Parents
Loading