fix: caching allocator behaviour for quantization. (#12172)
* fix: caching allocator behaviour for quantization.
* up
* Update src/diffusers/models/model_loading_utils.py
Co-authored-by: Aryan <aryan@huggingface.co>
---------
Co-authored-by: Aryan <aryan@huggingface.co>