Revert "Minor fixs to make torchbench runable on torch/xla (#107919)"
This reverts commit ed8f21282fca07621836a14f7d517148e1b944c3.
Reverted https://github.com/pytorch/pytorch/pull/107919 on behalf of https://github.com/izaitsevfb due to Conflicts with the revert of 106914 ([comment](https://github.com/pytorch/pytorch/pull/107919#issuecomment-1696662453))