Fix spectral_norm load_state_dict with strict=False (#22545)
Summary:
Fixes https://github.com/pytorch/pytorch/issues/21251
also fixes some missing hook removals.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/22545
Differential Revision: D16139506
Pulled By: soumith
fbshipit-source-id: 552a9f9f91be328a47ee8f1e1d29c1f59b0ebca3
