Fixing the enforcement of the zero_point
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/25193
Test Plan: Imported from OSS
Differential Revision: D17058781
Pulled By: zafartahirov
fbshipit-source-id: 7c665e1a0618c04a44f0c0e72e1bcc741a388e1c
