[PT-D][Tensor Parallelism][2/N] Sync TP API change to PT prod (#89467)
This is part of TP Beta Release efforts.
ref: https://github.com/pytorch/tau/issues/576
Pull Request resolved: https://github.com/pytorch/pytorch/pull/89467
Approved by: https://github.com/wanchaol