DeepSpeed
ZeRO-3: fix stale and lost gradients of unused parameters
#8981
Merged

ZeRO-3: fix stale and lost gradients of unused parameters #8981

xylian86
xylian86 ZeRO-3: do not apply a param's gradient from an earlier step again
e8bae05e
xylian86 xylian86 requested a review from tjruwase tjruwase 18 days ago
xylian86 xylian86 requested a review from tohtana tohtana 18 days ago
xylian86 xylian86 requested a review from loadams loadams 18 days ago
chatgpt-codex-connector
chatgpt-codex-connector
chatgpt-codex-connector commented on 2026-11-02
hwchen2017 hwchen2017 requested a review from copilot-pull-request-reviewer copilot-pull-request-reviewer 18 days ago
copilot-pull-request-reviewer
copilot-pull-request-reviewer commented on 2026-11-02
xylian86 ZeRO-3: fix the boundary fill on padding-only ranks and with safe_set
0e306672
xylian86 Merge branch 'master' into xinyu/zero3-unused-param-grads
7848a58b
0z5a
0z5a commented on 2026-11-03
sfc-gh-truwase
sfc-gh-truwase approved these changes on 2026-11-09
tohtana tohtana added needs-gpu-test
tohtana tohtana removed needs-gpu-test
PKUWZP Merge branch 'master' into xinyu/zero3-unused-param-grads
546909be
xylian86 xylian86 enabled auto-merge 2 days ago
xylian86 xylian86 merged daf175b1 into master 2 days ago
xylian86 xylian86 deleted the xinyu/zero3-unused-param-grads branch 2 days ago

Login to write a write a comment.

Login via GitHub

Assignees
No one assigned
Labels
Milestone