[FX] Speed up non-parameter tensor lookup (#47325)
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/47325
Test Plan: Imported from OSS
Reviewed By: zdevito
Differential Revision: D24715484
Pulled By: jamesr66a
fbshipit-source-id: 983eef6212ae95f5ddd3255adc8a585fb336074c
