Don't override `torch_dtype` and don't use when `quantization_config` is set (#11039)
* Don't use `torch_dtype` when `quantization_config` is set
* up
* djkajka
* Apply suggestions from code review
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>