refine disabled test (#60040)
Summary:
This is to refine:
https://github.com/pytorch/pytorch/pull/60029
Pull Request resolved: https://github.com/pytorch/pytorch/pull/60040
Reviewed By: ezyang
Differential Revision: D29147009
Pulled By: Krovatkin
fbshipit-source-id: 37e01ac6e8d6f7e6b5c517f7804704f9136a56f5
