Inverse Square Root LR Schedule (#657)
* Implement inverse square root with warmup scheduler v0
* inverse square root LR Schedule
* scheduler
* unit tests
* Update llmfoundry/optim/scheduler.py
Co-authored-by: Brian <23239305+b-chu@users.noreply.github.com>
* Update llmfoundry/optim/scheduler.py
Co-authored-by: Brian <23239305+b-chu@users.noreply.github.com>
* fixes for PR conversations
* format
* fix type hint
---------
Co-authored-by: Brian <23239305+b-chu@users.noreply.github.com>
Co-authored-by: cody <cody@mosaicml.com>
