[quant][graphmode] Remove unused patterns (#35385)
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/35385
Test Plan:
python test/test_jit.py
Imported from OSS
Differential Revision: D20655298
fbshipit-source-id: bc5eda2640a809adb55d3d645c65fb02a6f2f444
