fx quant: quick cleanup for model_device (#48906)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/48906
As titled, removing some code which is no longer
needed after refactors.
Test Plan:
CI
Imported from OSS
Reviewed By: jerryzh168
Differential Revision: D25363079
fbshipit-source-id: 9e4bcf63f4f1c2a2d3fb734688ba593d72495349