PR #2706 BF16 optimizer for BF16+ZeRO Stage 1

BF16 optimizer for BF16+ZeRO Stage 1 #2706

tjruwase merged 6 commits into master from jomayeri/bf16-zero1

BF16 optimizer only with ZeRO stage 1.

ec205afb

jomayeri requested a review from

jeffra 2 years ago

jomayeri requested a review from

tjruwase 2 years ago

jomayeri requested a review from

mrwyattii 2 years ago

Merge branch 'master' into jomayeri/bf16-zero1

2b6d3a66

Merge branch 'master' into jomayeri/bf16-zero1

db669591

Updating to grad accum of fp32 for BF16 ZeRO1 case.

d5fb37be

Merge branch 'master' into jomayeri/bf16-zero1

4182b581

Merge branch 'master' into jomayeri/bf16-zero1

7d9f05b5

tjruwase approved these changes on 2023-01-18

tjruwase merged 8d87c89e into master 2 years ago

mrwyattii deleted the jomayeri/bf16-zero1 branch 2 years ago

Reviewers

tjruwase

jeffra

mrwyattii

Assignees

No one assigned

Labels

None yet

Milestone

No milestone