Fix memory leak in Adam, Adagrad, RMSProp (#23125)
Summary:
As reported in LaurentMazare/tch-rs#76, the memory grows when weight_decay is present when using Adam. It applies the same fix in https://github.com/pytorch/pytorch/issues/23007 to Adam, Adagrad and RMSProp.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/23125
Differential Revision: D16402421
Pulled By: soumith
fbshipit-source-id: 59eb4bd81b8bd9e1a5f7c068ed841f70a4c38a80
