Revert "Allow ufunc OpInfos to have no reference (#82348)"
This reverts commit 566d734396eb77e1781b5cfa14390e02ed7b2409.
Reverted https://github.com/pytorch/pytorch/pull/82348 on behalf of https://github.com/peterbell10 due to This stack broke macos tests on trunk
