Remove Input Pointer Caching for XNNPack (#61959)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/61959
We no longer need to cache the Input Pointer as XNNPACK has implemented a more robust approach where indirection buffer does not need to be recalculated even if activation tensor pointer changes, as long as tensor dimensions are the same.
This reverses the changes in https://github.com/pytorch/pytorch/pull/42840/files
Reviewed By: kimishpatel
Differential Revision: D29777605
fbshipit-source-id: c1750538c17bce34f885c6f1bbb1f7164ebba25b
