Revert "Add support for directly passing symint to empty"
This reverts commit 05664a957e025826572034477d5125e779778032.
Reverted https://github.com/pytorch/pytorch/pull/79494 on behalf of https://github.com/ezyang due to conflicts with earlier diff that needs revert
