Ensure optimizer state references are cleared (#100282)
Fixes https://github.com/pytorch/pytorch/issues/100264
Pull Request resolved: https://github.com/pytorch/pytorch/pull/100282
Approved by: https://github.com/janeyx99, https://github.com/yanboliang
