Revert "Disable XLA test (#86123)" (#86151)
And also remove torch_patches/.torch_pin to mitigate the sev https://github.com/pytorch/pytorch/issues/86093 until XLA fixes the weird logic in https://github.com/pytorch/xla/blob/master/scripts/apply_patches.sh#L17-L18. Ticket cut to XLA https://github.com/pytorch/xla/issues/4068
Pull Request resolved: https://github.com/pytorch/pytorch/pull/86151
Approved by: https://github.com/kit1980
