[Quant] Remove weight from DTypeConfig for non-weighted ops
Summary: Weight dtypes should be specified only for weighted
ops like conv and linear. This commit removes weight dtypes
from the DTypeConfigs used in binary ops and fixed qparams ops.
Test Plan:
python test/test_quantization.py TestQuantizeFx
python test/test_quantization.py TestQuantizeFxOps
Reviewers: jerryzh168, vkuzo
Subscribers: jerryzh168, vkuzo
ghstack-source-id: aa352df1f52274bb345af29c3fc29d16669997aa
Pull Request resolved: https://github.com/pytorch/pytorch/pull/86335
