Numeric Suite: Swap with shadow modules only for quantized part of model (#51052)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/51052
Ensure that shadow modules are inserted only for quantized modules in a model. Removes redundant module insertion.
ghstack-source-id: 121041113
Test Plan: buck test caffe2/test:quantization -- 'test_compare_model_stub_partial \(quantization\.test_numeric_suite\.TestEagerModeNumericSuite\)'
Reviewed By: vkuzo
Differential Revision: D26054016
fbshipit-source-id: 73fc2fd2f0239b0363f358c80e34566d06a0c7cb
