[quant][graphmode][refactor] Remove reduandent code (#54073)
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/54073
Test Plan: Imported from OSS
Reviewed By: vkuzo
Differential Revision: D27086067
fbshipit-source-id: b1995138de56f1352c5df03378ebc2832bf35ef7
