DeepSpeed
add sharded checkpoint loading for AutoTP path to reduce the peak mem…
#3102

Merged

add sharded checkpoint loading for AutoTP path to reduce the peak mem… #3102

tjruwase merged 10 commits into deepspeedai:master from sywangyi:autoTP_memory_opt

sywangyi requested a review from

RezaYazdaniAminabadi 2 years ago

sywangyi requested a review from

jeffra 2 years ago

sywangyi requested a review from

mrwyattii 2 years ago

sywangyi requested a review from

awan-10 2 years ago

sywangyi requested a review from

cmikeh2 2 years ago

sywangyi requested a review from

arashb 2 years ago

sywangyi force pushed from b48734a4 to 7340a93e 2 years ago

add sharded checkpoint loading for AutoTP path to reduce the peak mem…

c78f5020

sywangyi force pushed from 7340a93e to c78f5020 2 years ago

sywangyi force pushed from 13dfd1c8 to 464db610 2 years ago

fix gptj sharded checkpoint loading problem

c40e911f

sywangyi force pushed from 464db610 to c40e911f 2 years ago