More element types in AllGather and AllToAll (#16941)
Two things done in this PR.
- [2nd commit] More tensor element types are supported because in
distributed computation, we need to re-shard tensors in many different
types.
- [1st commit] We now specify opset version in test models. Without this
change, those models will have opset=20 with latest ONNX and results
test errors.
- [3rd commit] Tests are modified to test `AllGather` and `AllToAll` for
boolean tensors. Several graph patterns are tried for tests. We found
that `int64_tensor -> Cast -> bool_tensor -> AllToAll -> bool_tensor ->
Cast -> int64_tensor` always generate random results. My guess is that
`AllToAll` needs to synchronize all GPUs before calling `ncclSend` and
`ncclRecv` since `AllGather` doesn't hit this problem. For reproducing
the error, search for `TODO` in this PR. Note that this PR doesn't fix
it.
