DeepSpeed
7054ba21 - Clear folding correction markers with consumed BF16 gradients

Commit
58 days ago
Clear folding correction markers with consumed BF16 gradients The immediate-update hook releases param.grad before the folding boundary reducer can clear its correction marker. Clear the marker when that gradient is consumed, leaving graph harvesting and epilogue accumulation unchanged. Add two-rank two-step and multi-micro-batch BF16 training coverage, including reentrant checkpointing and a public accumulated-gradient access example. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: yh0903 <helloyu0903@gmail.com>
Author
Parents
Loading