Skip stable diffusion models in torchbench, get tests and benchmarks green (#167896)
Summary:
X-link: https://github.com/pytorch/pytorch/pull/167896
Approved by: https://github.com/aorenste, https://github.com/shunting314
ghstack dependencies: #167609
Reviewed By: jeanschmidt
Differential Revision: D87148821
fbshipit-source-id: abe626d382e67cc8da67d4888cb1e6606d7e75ee