Add 'noarch' tests which only run in one CI config (#53747)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/53747
Fixes #53743
Signed-off-by: Edward Z. Yang <ezyang@fb.com>
Test Plan: Imported from OSS
Reviewed By: albanD
Differential Revision: D26971343
Pulled By: ezyang
fbshipit-source-id: cee7aa10063ae674f741406a3af830e4b4f128df
