PR #5170 Fix allreduce for BF16 and ZeRO0

Fix allreduce for BF16 and ZeRO0 #5170

tohtana merged 5 commits into master from tohtana/fix_bf16_z0_reduce

fix gradient clipping

794e9921

Merge branch 'master' into tohtana/fix_fp32_clipping

98714212

perform allreduce on FP32 when BF16 optimizer is enabled

84ced5be

ZZMerge branch 'master' into tohtana/fix_bf16_z0_reduce

fde1df02

tjruwase approved these changes on 2024-02-21

tohtana changed the title ~~Tohtana/fix bf16 z0 reduce~~ Fix allreduce for BF16 and ZeRO0 1 year ago

tohtana marked this pull request as ready for review 1 year ago

tohtana requested a review from

mrwyattii 1 year ago

Merge branch 'master' into tohtana/fix_bf16_z0_reduce

23df2cfa

tohtana enabled auto-merge 1 year ago

tohtana merged dd3690c5 into master 1 year ago

tohtana deleted the tohtana/fix_bf16_z0_reduce branch 1 year ago

Reviewers

tjruwase

mrwyattii

Assignees

No one assigned

Labels

None yet

Milestone

No milestone