Dont allow conversions with QInt.
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/22931
Test Plan: Imported from OSS
Differential Revision: D16467985
Pulled By: gchanan
fbshipit-source-id: 3925fc96a641e66b92fa65c542a2a23190c915a5
