PR #3622 zero3 performance optimizations

zero3 performance optimizations #3622

tjruwase merged 13 commits into deepspeedai:master from BacharL:perf1

Remove dead code

e1e4b753

Prevent evaluation of debug strings

e3dbb7a3

BacharL requested a review from

jeffra 2 years ago

BacharL requested a review from

tjruwase 2 years ago

BacharL requested a review from

samyam 2 years ago

BacharL requested a review from

mrwyattii 2 years ago

BacharL force pushed from 3b588087 to f8c7ad3a 2 years ago

BacharL force pushed from f8c7ad3a to a942dfa1 2 years ago

BacharL force pushed from a942dfa1 to 3a87dfa2 2 years ago

Use contiguous gradients tensor reduce scatter between ranks

bd4d724a

BacharL force pushed from 3a87dfa2 to d6a8711a 2 years ago

BacharL force pushed from d6a8711a to bd4d724a 2 years ago

move overflow tracker to optimizer.step

9cf826df

tjruwase commented on 2023-05-30