Cache mm padding decision (#102200)
Rebase of https://github.com/pytorch/pytorch/pull/100982 which was already accepted
Pull Request resolved: https://github.com/pytorch/pytorch/pull/102200
Approved by: https://github.com/ngimel, https://github.com/jansel
