.github: Temporarily disable xla test config (#70453)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/70453
Removes the current xla config, downstream `pytorch/xla` is broken for
clang compilation so temporarily removing this config until the xla team
can fix this upstream CI.
Context: https://github.com/pytorch/xla/pull/3255/files#r775980035
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
Test Plan: Imported from OSS
Reviewed By: zengk95
Differential Revision: D33338463
Pulled By: seemethere
fbshipit-source-id: 1ef332c685d5e2cc7e2eb038e93bd656847fd099
