DeepSpeed
500997a4 - Log the skipped optimizer step on gradient overflow (#8417)

Commit
8 days ago
Log the skipped optimizer step on gradient overflow (#8417) ZeRO returns early from `step()` on non-finite gradients. `DynamicLossScaler` reports that from inside `update_scale`, but `CreateLossScaler` only returns it for `fp16 + dynamic_scaling` — bf16 and fp32 always get the static `LossScaler`, whose `update_scale` is an inherited no-op. The skip is counted in `engine.skipped_steps`, but that is only surfaced by `_report_progress`, gated on `steps_per_print`, and `STEPS_PER_PRINT_DEFAULT` is `None`. So on a default bf16 config the loop advances, the LR scheduler steps and metrics are emitted while no weights are updated, with no output at all. Fixes #8415 --------- Signed-off-by: Abhishek Kulkarni <abhishek.kulkarni@snowflake.com> Co-authored-by: Olatunji Ruwase <tunji.ruwase@snowflake.com>
Parents
Loading