[quant][fx] Cleanup some unused states and args
Summary:
* Removed "patterns" from observed module since it's no longer needed
* Removed an arg from insert_observer
* Removed some unused keys in checking the validity of qconfig_dict
Test Plan:
python test/test_quantization.py TestQuantizeFx
python test/test_quantization.py TestQuantizeFxOps
python test/test_quantization.py TestQuantizeFxModels
Reviewers:
Subscribers:
Tasks:
Tags:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/75521
Approved by: https://github.com/andrewor14
