Have batch_size=1 whenever possible, add more exclusions (#1508)
Summary:
I had thought resetting the params and deleting the model would be enough, but it was not. Also added more exclusions for inductor issue https://github.com/pytorch/pytorch/issues/97361.
Will be running a test of the suite before landing this to ensure no other issues.
Pull Request resolved: https://github.com/pytorch/benchmark/pull/1508
Reviewed By: xuzhao9
Differential Revision: D44461785
Pulled By: janeyx99
fbshipit-source-id: b8e643de1376dba2e55d532bc3954483eb68f64d