[PyTorch] Avoid refcount bump in TensorArg (#54934)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/54934
It looks like the vast majority of usage is just borrowing a pre-existing Tensor.
ghstack-source-id: 125216052
Test Plan: Existing CI.
Reviewed By: hlu1
Differential Revision: D27415131
fbshipit-source-id: d5a8dc4ca5d48ca3eaa3664655b724094e61f371
