[ROCm] CI improvements for ROCm
Currently ROCm stream allocator has a couple of issues:
1. Even if we select the platform allocator from JAX, XLA actually
provides the default BFC allocator, but it has a wiring issue with
registering to the right stream. This causes occasional flaky tests
due to a kernel in flight reading/writing memory that is reused by
another kernel.
2. The stream allocator for ROCm currently hangs whenever all workers
get conv-related work (MIOpen). The issue is in XLA itself and a fix
is in progress.
Because of these issues we want to give the default allocator a go.
Downstream testing has been promising.
Also dropped the explicit --dist=loadfile so pytest-xdist falls back to
its default (load), matching what the CUDA CI uses, since loadfile was a
little slower.