[vulkan] Fix asserts (#61495)
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/61495
Test Plan: Imported from OSS
Reviewed By: SS-JIA
Differential Revision: D29647357
Pulled By: IvanKobzarev
fbshipit-source-id: cb4ba15f28625ea6e667883c9a2d31eba48b6f37
