DeepSpeed
b33873d2 - Fix hpZ with zero element (#5652)

Commit
1 year ago
Fix hpZ with zero element (#5652) Fix corner cases where hpz secondary partition has zero element. This ensure that `sec_numel` is at least zero. For this scenario, copying is really not necessary except that all ranks need to synchronize at the end of secondary partition. This is a good solution until [2nd tensor all-gather vs 2nd tensor partition issue ](https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/runtime/zero/partition_parameters.py#L1706) is properly fixed. Fixes: #5642 --------- Co-authored-by: Logan Adams <loadams@microsoft.com> Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
Author
Parents
Loading