Some small performance fixes for c10 dispatcher (#20472)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/20472
ghimport-source-id: d118bf8d48eea3faf241a7288fcad1bb6a5f051f
Differential Revision: D15332284
Pulled By: li-roy
fbshipit-source-id: a8d9e50a440a7ad3ee730f70c0fcae06ae848cbd
