Reduce number of `prim::profile` (#43147)
Summary:
Fixes #{issue number}
Pull Request resolved: https://github.com/pytorch/pytorch/pull/43147
Reviewed By: colesbury
Differential Revision: D23190137
Pulled By: Krovatkin
fbshipit-source-id: bf5f29a76e5ebfb5b9d3b6adee424e213c25891b
