DeepSpeed
3fd762cf - Fix WarmupCosineLR multi-group initialization (#7969)

Commit
87 days ago
Fix WarmupCosineLR multi-group initialization (#7969) `WarmupCosineLR` returned a singleton pre-start LR list even when the optimizer had multiple parameter groups. Because scheduler initialization applies LRs with `zip(param_groups, lrs)`, only group 0 was updated and later groups kept their base LR before the first optimizer step. The fix changes the pre-start scheduler outputs to match the multi-group contract by returning scalar `0.0` from `get_lr_ratio()` and a zero-filled LR list sized to `self.org_lrs`. Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Author
Parents
Loading