DeepSpeed
573ce488 - Skip zero-sized parameters in HP fragment mapping (#8298)

Commit
3 days ago
Skip zero-sized parameters in HP fragment mapping (#8298) ## Summary - exclude zero-sized parameters from ZeRO-1/2 HP-fragment overlap mapping - avoid constructing an empty fragment when an empty parameter's offset lies inside a partition Fixes #8297. ## Testing - current `master` (`edaa722`), 2x A6000: minimized ZeRO-1 reproducer fails before the change with `fragment start 8 should be < fragment_end 8` and completes after the change - 2-process CPU/Gloo: minimized reproducer completes with both ZeRO-1 and ZeRO-2 after the change - the trailing zero-sized-parameter reproducer from #8279 still completes - `python3 -m py_compile deepspeed/utils/mixed_precision_linkage.py` Signed-off-by: Cao Yuhang <caoyuhang@fwerkor.com> Co-authored-by: Olatunji Ruwase <tunji.ruwase@snowflake.com>
Author
Parents
Loading