Revert "Skip nonzero unbacked SymInt memo in inference mode (#122147)"
This reverts commit 5e2687391229cee6e4dc0214f9208b4ecbe058c1.
Reverted https://github.com/pytorch/pytorch/pull/122147 on behalf of https://github.com/jeanschmidt due to Reverting to see if trunk error in inductor are related ([comment](https://github.com/pytorch/pytorch/pull/122147#issuecomment-2007513000))
