Megatron-DeepSpeed
Fix mixed fused layer norm to mimick nn.LayerNorm for torch>1.11
#281
Merged
Overview
Commits
14
Changes
View On GitHub
Commits
  • If pytorch>1.11 available we can use nn.LayerNorm instead of MixedLayerNorm
    thomasw21 committed 4 years ago
  • Add MixedFusedLayerNorm fix
    thomasw21 committed 4 years ago
  • Woops
    thomasw21 committed 4 years ago
  • Convert weight/bias only once
    thomasw21 committed 4 years ago
  • Revert "Convert weight/bias only once"
    thomasw21 committed 4 years ago
  • Turns out LayerNorm for bf16 is slower using torch==1.11
    thomasw21 committed 4 years ago
  • Woops
    thomasw21 committed 4 years ago
  • Rewrite if condition
    thomasw21 committed 4 years ago
  • Use version package instead
    thomasw21 committed 4 years ago
  • Test for LayerNorm
    thomasw21 committed 4 years ago
  • Improve test to use torch_assert_equal + minor fixes
    thomasw21 committed 4 years ago
  • Force bfloat16
    thomasw21 committed 4 years ago
  • Woops
    thomasw21 committed 4 years ago
  • Fix torch version comparison
    thomasw21 committed 4 years ago
FAQ Terms Privacy Refunds Impressum
Loading