Implementation of ConstantThenLinearWarmupLRPolicy and CompositeCyclicalLRPolicy (#25970)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/25970
ConstantThenLinearWarmupLRPolicy:
* first use a constant warm up
* then ramp up to the fixed learning rate linearly
CompositeCyclicalLRPolicy:
* first use a constant warm up
* then ramp up to the fixed learning rate linearly
* then use cyclical learning rates for the rest of time
Pull Request resolved: https://our.intern.facebook.com/intern/opensource/shipit/preview/D17302632/
Test Plan:
* buck test
* https://our.intern.facebook.com/intern/testinfra/testconsole/testrun/5910974518377039/
* https://our.intern.facebook.com/intern/testinfra/testrun/1407375027118303
* checked the consistency of learning rates w.r.t. iterations with offline simulations n143987
Reviewed By: swatirallapalli
Differential Revision: D17302632
fbshipit-source-id: 1098d4dd9109a48932b76e36d78239e49f8077a1
