DeepSpeed
18489835 - fix gradient accumulation for z2+offload

Commit

2 years ago

fix gradient accumulation for z2+offload

References

#6550 - Fix gradient accumulation for Z2+offload

#6554 - Improve consistency of zero_grad

Author

Masahiro Tanaka

Parents

Loading