Fix a subset of issues mentioned in #1567 (#1573)
Summary:
Fixes the non memory related issues in https://github.com/pytorch/benchmark/actions/runs/4770804314
Pull Request resolved: https://github.com/pytorch/benchmark/pull/1573
Reviewed By: xuzhao9
Differential Revision: D45232801
Pulled By: janeyx99
fbshipit-source-id: 79934c068a2607571c23fcf5b4a1dcdeaed6a497
