Remove redundant queries of qconfig in `insertObservers` (#31292)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/31292
att
Also we need to do this check after we call `insertObservers` on invoked modules
as well since qconfig can be None for parent module while being valid for invoked modules
Test Plan:
.
Imported from OSS
Differential Revision: D19146668
fbshipit-source-id: be6811353d359ed3edd5415ced29a4999d86650b
