Only create a new clone of observer when we actually insert it.
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/25931
Test Plan: Imported from OSS
Differential Revision: D17288546
Pulled By: ZolotukhinM
fbshipit-source-id: 01584d27c0ebd127b845e560aedd1f1d9d298c5e
Author
Mikhail Zolotukhin
