DeepSpeed
ed090241 - Release each BF16 gradient once immediate_grad_update has accumulated it

Commit
58 days ago
Release each BF16 gradient once immediate_grad_update has accumulated it With immediate_grad_update, BF16_Optimizer adds each parameter's BF16 gradient into its FP32 accumulation buffer as soon as autograd finishes it, in a hook named accumulate_hp_grads_and_remove_lp, but it never removed the BF16 gradient. #5328 zeroed it there and #6508 reverted that because zeroing frees nothing. So every BF16 gradient stays alive beside the FP32 buffer through the end of backward and the gradient all-reduce, 6.7 GB per rank for Qwen3-30B-A3B at EP16, where an allocator trace puts the step's peak once the loss is chunked. Reduction, clipping and the step read the FP32 buffer, so the hook now drops the BF16 gradient after accumulating it, except under graph harvesting, which replays kernels on fixed gradient addresses. A training test checks that immediate accumulation gives bit-identical FP32 gradients and parameters to epilogue accumulation over two micro-batches, with no BF16 gradient left after backward. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: yh0903 <helloyu0903@gmail.com>
Author
Committer
Parents
Loading