DeepSpeed
Fix a convergence issues in TP topology caused by incorrect grad_norm.
#5411

Merged

Fix a convergence issues in TP topology caused by incorrect grad_norm. #5411

conglongli merged 9 commits into deepspeedai:master from inkcherry:tp_grad_fix

fix grad norm for tp

287fa5eb

refine code

a7e8a7fe

remove unnecessary clip_gradients fun

ea41928e

improve perf by loop-free implementations

e74b7ca6

Modify the comments.

79cc4cef

update

3ebed5ea

Merge remote-tracking branch 'master' into tp_grad_fix

fc537b8e

inkcherry requested a review from

mrwyattii 1 year ago

inkcherry requested a review from

tjruwase 1 year ago

tjruwase removed review request from

mrwyattii 1 year ago

tjruwase requested a review from

tohtana 1 year ago

tjruwase requested a review from

conglongli 1 year ago

conglongli requested changes on 2024-04-15

refine comments

df976ca6

tohtana approved these changes on 2024-04-16

Merge branch 'master' into tp_grad_fix

a40263f5

conglongli enabled auto-merge 1 year ago

conglongli approved these changes on 2024-04-16

conglongli merged 0896503e into master 1 year ago

Reviewers

tohtana

conglongli

tjruwase

Assignees

No one assigned

Labels

None yet

Milestone

No milestone