PR #272 sync layer norms

sync layer norms #272

stas00 merged 14 commits into main from layer-norm-auto-sync

sync layer norms

9ba8210f

all_reduce is an in_place operation

fbd47eed

Make dataloader use another random generator (#276)

a9fb317e

do all_reduce op.AVG directly

8c1ed225

Merge remote-tracking branch 'origin/main' into layer-norm-auto-sync

8937dede

stas00 requested a review 3 years ago

add eval dataloader deadlock workaround

b015ec15

thomasw21 commented on 2022-05-24

revert generator sync

10f50184

Merge remote-tracking branch 'origin/main' into layer-norm-auto-sync

5be13991

Merge remote-tracking branch 'origin/main' into layer-norm-auto-sync

dedf15c6

make auto-sync configurable; basic test; cleanup

76f2fd7c

Merge remote-tracking branch 'origin/main' into layer-norm-auto-sync

3070d61c

test with updated AMI image

e2ffb5e9

fix unrelated test

44bc82ef

Merge remote-tracking branch 'origin/main' into layer-norm-auto-sync

0030e694

stas00 merged e1c479e5 into main 3 years ago

stas00 deleted the layer-norm-auto-sync branch 3 years ago

Reviewers

thomasw21

Assignees

No one assigned

Labels

None yet

Milestone

No milestone