transformers
45bd05b8 - Init the process group with a load-scaled timeout for sharded loading (#48228)

Commit
77 days ago
Init the process group with a load-scaled timeout for sharded loading (#48228) * Init the process group with a load-scaled timeout for sharded loading Sharded loading of a TB-scale checkpoint takes tens of minutes and ranks finish far apart (measured: >10 minutes spread on GLM-4.6, 714 GB over 16 ranks), so the fastest ranks' first collective dies on the default 10-minute NCCL watchdog. Use a 2-hour timeout when transformers initializes the group itself. * Move the timeout comment onto the argument it explains
Author
Parents
Loading