DeepSpeed
991ccf0f - Keep invalid group-norm sentinels from becoming a finite 1.0 step (#8638)

Commit
43 days ago
Keep invalid group-norm sentinels from becoming a finite 1.0 step (#8638) ## Summary - DeepSpeed encodes inf/NaN group norms as `-1`. Combining those with `vector_norm` squares the sentinel to `1.0`, so clipping and Adam treat a failed group as healthy and apply an unclipped step. - Treat non-finite or negative group norms as overflow (`inf`), skip the optimizer step, reset overflow each step, and advance the loss scaler once (including ZeRO-3 / SuperOffload / ZenFlow / FP16 fused+unfused / BF16). - Direct `unscale_and_clip_grads` callers map the sentinel to `inf` so they still zero gradients instead of using clip_coef=1. This is split out of the ZeRO-2 oversized-gradient / stream-lifetime work so the sentinel contract can land independently. ## Test plan - [x] CPU, rebased onto current master (`75eb961d1`, includes #8730): 17 passed in `tests/unit/v1/zero/test_invalid_grad_norm.py` (`combine_grad_norm_groups` / `get_global_norm` / `CheckOverflow`, ZeRO-1 clip, CPU-offload combine, ZeRO-3 / BF16 / SuperOffload overflow paths) - [x] GPU (2× H200, prior run): `TestZeROBFloat16Stage3InvalidNormRecovery`. Skipped on the rebase machine (2 GPUs required, 1 available) - [ ] CI unit tests for the touched runtime files, started by the rebase push --------- Signed-off-by: tunji-ruwase_snow <tunji.ruwase@snowflake.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Parents
Loading