Ensure that histogram observers have zero-point of zero for post ReLU activations (#37107)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/37107
Currently histogram observers relax both the min and max values of the activations for performance speedup reasons. This causes an issue for glow where there is a slow down if the zero-point is not zero for post ReLU activations.
ghstack-source-id: 102768017
Test Plan: buck test caffe2/test:quantization -- 'test_histogram_observer_one_sided \(quantization\.test_quantization\.RecordHistogramObserverTest\)' --print-passing-details
Differential Revision: D21187636
fbshipit-source-id: 8d616b9e9caf2979a26a215e99434f71025e3d8b
