Megatron-DeepSpeed
Test different layer norm
#270
Open
Overview
Commits
27
Changes
View On GitHub
Commits
  • WIP
    thomasw21 committed 4 years ago
  • Wip
    thomasw21 committed 4 years ago
  • Woops
    thomasw21 committed 4 years ago
  • WIP
    thomasw21 committed 4 years ago
  • Woops
    thomasw21 committed 4 years ago
  • Woops
    thomasw21 committed 4 years ago
  • Woops
    thomasw21 committed 4 years ago
  • Test with alibi
    thomasw21 committed 4 years ago
  • Still trying to reproduce
    thomasw21 committed 4 years ago
  • Huh
    thomasw21 committed 4 years ago
  • Have high LR to see weights actually change
    thomasw21 committed 4 years ago
  • Launch bf16
    thomasw21 committed 4 years ago
  • Woops
    thomasw21 committed 4 years ago
  • Make test to work with both bf16 and fp16 to see who fails
    thomasw21 committed 4 years ago
  • Woops
    thomasw21 committed 4 years ago
  • Remove assert
    thomasw21 committed 4 years ago
  • Try to figure out how the divergence happens
    thomasw21 committed 4 years ago
  • I think bias starts to diverge first
    thomasw21 committed 4 years ago
  • Woops
    thomasw21 committed 4 years ago
  • Woops
    thomasw21 committed 4 years ago
  • Woops
    thomasw21 committed 4 years ago
  • Add embed layer norm
    thomasw21 committed 4 years ago
  • Woops
    thomasw21 committed 4 years ago
  • Backward compatibility on torch
    thomasw21 committed 4 years ago
  • Better
    thomasw21 committed 4 years ago
  • Merge remote-tracking branch 'origin/main' into thomas/test_different_layer_norm
    stas00 committed 4 years ago
  • fix
    stas00 committed 4 years ago
FAQ Terms Privacy Refunds Impressum
Loading