Megatron-DeepSpeed
c20c8ba4
- Have high LR to see weights actually change
Go
Login via GitHub
Home
Pricing
FAQ
Install
Login
via GitHub
Commit
View On
GitHub
Commit
3 years ago
Have high LR to see weights actually change
References
#270 - Test different layer norm
#271 - Sync layer norm
#274 - Sync 4 layer norms - bf16, fp32, optimizer states on restart
Author
thomasw21
Parents
42d6b4e3
Loading