do all_reduce op.AVG directly - SemanticDiff

Commit

3 years ago

do all_reduce op.AVG directly

References

#272 - sync layer norms

#292 - a branch combining layer-norm-auto-sync and ds_ckpt_reshape

Author

stas00

stas00

Parents

Loading