[reland][quant][graphmode] Support a new category of ops in graph mode quantization (#37936)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/37936
Previously we classify ops like average pool to the category that doesn't require observation and
the quantization of these ops are done by swapping with dequantize ops: https://github.com/pytorch/pytorch/pull/33481
However, this operation is done in finalize, which means finalize is a numerics changing pass when we swap dequantize with
ops like average pool, this is not ideal since we want to restrict the scope of numerics changing passes.
Because although average pool doesn't require observation, quantized average pool = dequant + float32 average pool + quant
and swapping average pool with dequantize is a numerics changing operation.
This PR implements the support for that. We'll classify ops like average pool to a new category and we'll get average pool through fusion, like we did for other quantized ops. And the numerics changing pass will only happen in insert quant dequant pass, so the model will have the same numerics before and after finalize. With the new category, the debug only option(the model before finalize) for quantize_script will actually produce a model that's numerically consistent with the finalized model.
Test Plan: python test/test_quantization.py TestQuantizeScriptJitPasses
Differential Revision: D21432871
Pulled By: jerryzh168
fbshipit-source-id: 4926890441e39af4e459376038563c3882cc4c46
