bigscience-workshop/Megatron-DeepSpeed

Pull Requests Commits
already part of the function
stas00 committed 3 years ago
Verified 835a3e5c
bf16 comms requite pt-1.11
stas00 committed 3 years ago
Verified 70f91f82
Merge remote-tracking branch 'origin/main' into thomas/fix_layer_norm
stas00 committed 3 years ago
dbb59140
`torch.testing.assert_equal` didn't make it (#273)
stas00 committed 3 years ago
Verified 87a9dba0
use the bf16 branch for testing
stas00 committed 3 years ago
279a77eb
import mpu
stas00 committed 3 years ago
Verified 98d0e7cc
Force synchronize the layer norms parameters across all TP
thomasw21 committed 3 years ago
391ed488
Better
thomasw21 committed 3 years ago
07ccb3db
Backward compatibility on torch
thomasw21 committed 3 years ago
f0d6d179
Woops
thomasw21 committed 3 years ago
0cf35ee3
Add embed layer norm
thomasw21 committed 3 years ago
2389bfdf
Woops
thomasw21 committed 3 years ago
8ffb278f
Woops
thomasw21 committed 3 years ago
39d4b8f9
Woops
thomasw21 committed 3 years ago
311e5317
I think bias starts to diverge first
thomasw21 committed 3 years ago
7145f6df
Try to figure out how the divergence happens
thomasw21 committed 3 years ago
a5e32958
Remove assert
thomasw21 committed 3 years ago
6b19339c
Woops
thomasw21 committed 3 years ago
a0c09132
Make test to work with both bf16 and fp16 to see who fails
thomasw21 committed 3 years ago
5fbe1072
Woops
thomasw21 committed 3 years ago
a4172bf9
Launch bf16
thomasw21 committed 3 years ago
7f2441ed
Have high LR to see weights actually change
thomasw21 committed 3 years ago
c20c8ba4
Huh
thomasw21 committed 3 years ago
42d6b4e3
Still trying to reproduce
thomasw21 committed 3 years ago
02365d14
Test with alibi
thomasw21 committed 3 years ago
ce02dd16
Woops
thomasw21 committed 3 years ago
f152e487
Woops
thomasw21 committed 3 years ago
1f2f8007
Woops
thomasw21 committed 3 years ago
7fcff06b
WIP
thomasw21 committed 3 years ago
29372806
Woops
thomasw21 committed 3 years ago
1cdcd7de
Terms Of Use Privacy Policy Refund Policy Impressum