Fix TorchAO related bugs; revert device_map changes (#10371)
* Revert "Add support for sharded models when TorchAO quantization is enabled (#10256)"
This reverts commit 41ba8c0bf6b3dc3ebd0fa6b96ecf671fa4171566.
* update tests
* udpate
* update
* update
* update device map tests
* apply review suggestions
* update
* make style
* fix
* update docs
* update tests
* update workflow
* update
* improve tests
* allclose tolerance
* Update src/diffusers/models/modeling_utils.py
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* Update tests/quantization/torchao/test_torchao.py
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* improve tests
* fix
* update correct slices
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
