FIX Multiple PEFT errors after v5 transition (#44592)
Multiple PEFT tests are failing due to recent changes in transformers.
- hf_device_map attribute may not exist in some cases
- respect inference_mode in load_adapter
- new model loading requires changes for bnb (SCB attribute etc.)
These tests have been failing for quite some time now but nobody
noticed.
