DeepSpeed
080eb6b0 - Fix ZeRO parameter alignment for grouped_mm (#8277)

Commit
37 days ago
Fix ZeRO parameter alignment for grouped_mm (#8277) ## Summary ZeRO stage 1/2 stores model parameters as views into flattened fp16/bf16 buffers. Individual parameter views can start at non-16-byte offsets even when the flat buffer itself is aligned, which breaks alignment-sensitive kernels such as `torch._grouped_mm`. Pad parameter boundaries inside the ZeRO flat buffer so model parameter views remain 16-byte aligned without duplicating misaligned parameters. The padded layout is propagated through partition/gradient offsets, LP↔HP linkage, DeepCompile gradient buffers, checkpoint restore, and `zero_to_fp32` reconstruction. Older checkpoints without parameter-alignment padding remain loadable; their compact layout is converted when restored. ## Tests - ZeRO-1/2 BF16 regression with a deliberately misaligned parameter layout. - Verifies zero-copy aligned flat-buffer views across optimizer steps. - Verifies checkpoint reload with `load_module_only=True` and `load_optimizer_states=False`. - Synced with current `master`, including the ZeRO-1/2 DeepCompile file rename. Fixes #8276 --------- Signed-off-by: Cao Yuhang <caoyuhang@fwerkor.com> Signed-off-by: Masahiro Tanaka <tanaka.masahiro@gmail.com> Co-authored-by: Masahiro Tanaka <tanaka.masahiro@gmail.com> Co-authored-by: Masahiro Tanaka <81312776+tohtana@users.noreply.github.com>
Author
Parents
Loading