Fix mismatching loading in from_pretrained with/without accelerate (#28414)
* fix mismatching behavior in from_pretrained with/without accelerate
* meaningful refactor
* remove added space
* add test
* fix model on the hub
* comment
* use tiny model
* style