Megatron-DeepSpeed
Fix mixed fused layer norm to mimick nn.LayerNorm for torch>1.11
#281
Merged
Overview
Commits
14
Changes
View On GitHub

Fix mixed fused layer norm to mimick nn.LayerNorm for torch>1.11 #281

thomasw21 merged 14 commits into main from thomas/remove_mixed_fused_layer_norm
thomasw21
thomasw21 thomasw21 requested a review 3 years ago
thomasw21 If pytorch>1.11 available we can use nn.LayerNorm instead of MixedLay…
26bd3d2b
thomasw21 thomasw21 force pushed from 7a3a32ac to 26bd3d2b 3 years ago
thomasw21 Add MixedFusedLayerNorm fix
79922d4a
thomasw21 Woops
27991b8d
thomasw21 thomasw21 requested a review from stas00 stas00 3 years ago
thomasw21 Convert weight/bias only once
c4b05ee9
thomasw21 Revert "Convert weight/bias only once"
5db805c3
thomasw21 Turns out LayerNorm for bf16 is slower using torch==1.11
824b9c5f
thomasw21 Woops
febce3ca
thomasw21 Rewrite if condition
400ec42f
thomasw21
stas00
stas00 commented on 2022-04-30
thomasw21 Use version package instead
f7d4e779
thomasw21 Test for LayerNorm
2510497d
thomasw21 Improve test to use torch_assert_equal + minor fixes
99867bd0
thomasw21 Force bfloat16
d84fc727
thomasw21 Woops
723489bb
thomasw21
thomasw21 thomasw21 changed the title Remove mixed fused layer norm in favor of nn.LayerNorm Fix mixed fused layer norm to mimick nn.LayerNorm for torch>1.11 3 years ago
thomasw21 thomasw21 requested a review from stas00 stas00 3 years ago
stas00
stas00 commented on 2022-05-02
stas00
stas00 approved these changes on 2022-05-03
thomasw21 Fix torch version comparison
37500d97
thomasw21 thomasw21 merged 908dc9cb into main 3 years ago
thomasw21 thomasw21 deleted the thomas/remove_mixed_fused_layer_norm branch 3 years ago

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum