Megatron-DeepSpeed
c7f20066
- Sync lp/hp/optim for layer norms
Go
Login via GitHub
Home
Pricing
FAQ
Install
Login
via GitHub
Commit
View On
GitHub
Commit
3 years ago
Sync lp/hp/optim for layer norms
References
#274 - Sync 4 layer norms - bf16, fp32, optimizer states on restart
Author
tjruwase
Parents
a5b5edc0
Loading