Revert "enable fx graph cache on torchbench (#128239)"
Summary:
This reverts commit 734e8f6ad7e7f0fa0341fb658f1f986225173f5f.
Reverted https://github.com/pytorch/pytorch/pull/128239 on behalf of https://github.com/huydhn due to Sorry for reverting your change but it seems to surface a bunch of inductor failures in trunk https://hud.pytorch.org/pytorch/pytorch/commit/734e8f6ad7e7f0fa0341fb658f1f986225173f5f ([comment](https://github.com/pytorch/pytorch/pull/128239#issuecomment-2159789242))
Reviewed By: clee2000
Differential Revision: D58444069
fbshipit-source-id: 031136895b2a19ce20ae3c5c74ec43d925acb218
