[Vulkan] Fix seg fault during descriptor set allocation on some platforms (#57825)
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/57825
Test Plan: Imported from OSS
Reviewed By: kimishpatel
Differential Revision: D28294939
Pulled By: SS-JIA
fbshipit-source-id: a7847fdc118ce96bfee83e755888e28dd605c1fb
