Revert "Add symbolic singleton int (#110370)"
This reverts commit a7145cb3a42e925209c7f34c0b8b169dc72ff4c6.
Reverted https://github.com/pytorch/pytorch/pull/110370 on behalf of https://github.com/PaliC due to bottom diff is causing a plethora of internal failures ([comment](https://github.com/pytorch/pytorch/pull/110370#issuecomment-1749801188))
