bigscience-workshop/Megatron-DeepSpeed

Pull Requests Commits

Test out the loss from the fp32 weights and optimizer states

thomasw21 committed 3 years ago

189f0547

I'm surprised one can apply inplace operation here

thomasw21 committed 3 years ago

c3844b5c

thomasw21 committed 3 years ago

007ecb4b

Fix non matching torch_rng_state for tp_rank=0

thomasw21 committed 3 years ago

2c49216a

Test that torch_rng_state

thomasw21 committed 3 years ago

7ebbed16

Test for final weight

thomasw21 committed 3 years ago

1fc4fe82

Figure out why state differs

thomasw21 committed 3 years ago

e0ef1683

New list of matching embeddings

thomasw21 committed 3 years ago

42997b2a

Row column bias should be synchronized as well

thomasw21 committed 3 years ago

89ab0b72

thomasw21 committed 3 years ago

ce017338

Compare model_states_rank

thomasw21 committed 3 years ago

0b7afcc9

Now that we have a force sync mechanism, let's try to reproduce

thomasw21 committed 3 years ago

64a62c80

thomasw21 committed 3 years ago

77abee61

Test that force sync will allow TP changes

thomasw21 committed 3 years ago

09cead38

Fix checkpoint path

thomasw21 committed 3 years ago

e9431002

thomasw21 committed 3 years ago

04e89d14

thomasw21 committed 3 years ago

4574ec97

thomasw21 committed 3 years ago

7cdb1be8

thomasw21 committed 3 years ago

cc7ff45b

thomasw21 committed 3 years ago

5fb231c1

thomasw21 committed 3 years ago

d357839d

thomasw21 committed 3 years ago

2ab8a3ac

thomasw21 committed 3 years ago

55e10c63

thomasw21 committed 3 years ago

05c99db6

thomasw21 committed 3 years ago

3bc58243

thomasw21 committed 3 years ago

4b7207b5

Make a test that TP reshaping works

thomasw21 committed 3 years ago

b418b47a

all_reduce is an in_place operation

thomasw21 committed 3 years ago

8271d419

run just the test we want for now

stas00 committed 3 years ago

3ec65f7c

reproduce the crashing on resume

stas00 committed 3 years ago

37795a92

Older