Change condition in swap module
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/23561
Test Plan:
python test/test_quantization.py
Imported from OSS
Differential Revision: D16570928
Pulled By: jerryzh168
fbshipit-source-id: 70f36f577ac657d015f3d7738819867742088e5a
