Only optimize after change (redo) (#59809)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/59809
Some how this didnt get landed previously in ghstack mixup
Test Plan: Imported from OSS
Reviewed By: pbelevich, zhxchen17
Differential Revision: D30200060
Pulled By: eellison
fbshipit-source-id: 47f256421a1fe1a005cd11fcc4d7f023b5990834
