DeepSpeed
f4802aa5 - Changing the order of pre_module hook registration so that the step_id is reset at the beginning of forward and not after the fetching of parameters of the first module. Partitioning parameters after GatheredParameters exit regardless of whether parameters are modified or not

Commit

4 years ago

Changing the order of pre_module hook registration so that the step_id is reset at the beginning of forward and not after the fetching of parameters of the first module. Partitioning parameters after GatheredParameters exit regardless of whether parameters are modified or not

Author

samyam

Parents

a3cee49b

DeepSpeed f4802aa5 - Changing the order of pre_module hook registration so that the step_id is reset at the beginning of forward and not after the fetching of parameters of the first module. Partitioning parameters after GatheredParameters exit regardless of whether parameters are modified or not

DeepSpeed
f4802aa5 - Changing the order of pre_module hook registration so that the step_id is reset at the beginning of forward and not after the fetching of parameters of the first module. Partitioning parameters after GatheredParameters exit regardless of whether parameters are modified or not