FIX Failing single GPU tests related to hotswap (#2385)
After unblocking single GPU tests with #2380, a couple of tests related
to hotswapping failed. This PR should (hopefully) address this.
1. Wrong error type caught with xfail
I set the wrong error type for the xfailing compiled hotswap diffusers
tests. This was because I hadn't checked out diffusers main when I was
checking locally.
2. Loosen tolerance
Some tests fail because an allclose does not match even though the
numbers in the logs look pretty much identical:
https://github.com/huggingface/peft/actions/runs/13404117333/job/37440752790#step:6:1929
This is most likely a problem with tolerances being too strict.
Unfortunately, I can't reproduce the error locally, so I have to guess
that moving from 1e-5 to 1e-4 will the issue.
