Fix NaN error in dynamic quantization in qLinear, re-enable test_quantized_rnn (#36009)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/36009
When scale is very small (less than float eps, but greater than minimum double precision value), computation of reciprocal of scale in floating point precision within FBGEMM returns inf, while QuantUtils does not. Changed computation in QuantUtils to occur with floating point precision to re-enable tests.
ghstack-source-id: 102896302
Test Plan:
buck test caffe2/test:quantization -- 'test_quantized_rnn \(quantization\.test_quantization\.PostTrainingDynamicQuantTest\)' --print-passing-details --run-disabled
Summary (total time 59.91s):
PASS: 1
FAIL: 0
SKIP: 0
FATAL: 0
TIMEOUT: 0
OMIT: 0
Differential Revision: D20853000
fbshipit-source-id: 948a888f5516b3ba9c6efb7de31ef2cc9d431991
