Tweak Vulkan memory use. (#47728)
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/47728
Test Plan: Imported from OSS
Reviewed By: SS-JIA
Differential Revision: D25032740
Pulled By: AshkanAliabadi
fbshipit-source-id: 7eb72538dc1aa3feb4e2f8c4ff9c675eb8e97057
