Remove observers in the end (#31407)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/31407
Remove observers in the end instead of before quantize tensor
since we still need them to find the quantization paramters for each module instance
Test Plan:
.
Imported from OSS
Differential Revision: D19162367
fbshipit-source-id: f817af87183f6c42dc97becea85ddeb7e050e2b1
