benchmark

b3328994 - Enhance cpu userbenchmark deal with failures and timeout (#1635)

Enhance cpu userbenchmark deal with failures and timeout (#1635) Summary: Fix cpu userbenchmark issue for not implemented models, and add timeout parameter for cpu userbenchmark It also works for roadmap https://github.com/pytorch/benchmark/issues/1293 For example: ```shell $ python run_benchmark.py cpu -m resnet50,Background_Matting,resnet18 --timeout 10 Running benchmark: ['/localdisk/chuanqiw/miniconda3/envs/torchdynamo/bin/python', '/localdisk/chuanqiw/PT/benchmark/userbenchmark/cpu/run_config.py', '-m', 'resnet50', '--device', 'cpu', '-t', 'eval', '-o', '/localdisk/chuanqiw/PT/benchmark/.userbenchmark/cpu/cpu-20230508013240'] Command '['/localdisk/chuanqiw/miniconda3/envs/torchdynamo/bin/python', '/localdisk/chuanqiw/PT/benchmark/userbenchmark/cpu/run_config.py', '-m', 'resnet50', '--device', 'cpu', '-t', 'eval', '-o', '/localdisk/chuanqiw/PT/benchmark/.userbenchmark/cpu/cpu-20230508013240']' timed out after 10 seconds Running benchmark: ['/localdisk/chuanqiw/miniconda3/envs/torchdynamo/bin/python', '/localdisk/chuanqiw/PT/benchmark/userbenchmark/cpu/run_config.py', '-m', 'Background_Matting', '--device', 'cpu', '-t', 'eval', '-o', '/localdisk/chuanqiw/PT/benchmark/.userbenchmark/cpu/cpu-20230508013240'] Running TorchBenchModelConfig(name='Background_Matting', device='cpu', test='eval', batch_size=None, jit=False, extra_args=[], extra_env=None) ... [NotImplemented] Running benchmark: ['/localdisk/chuanqiw/miniconda3/envs/torchdynamo/bin/python', '/localdisk/chuanqiw/PT/benchmark/userbenchmark/cpu/run_config.py', '-m', 'resnet18', '--device', 'cpu', '-t', 'eval', '-o', '/localdisk/chuanqiw/PT/benchmark/.userbenchmark/cpu/cpu-20230508013240'] Running TorchBenchModelConfig(name='resnet18', device='cpu', test='eval', batch_size=None, jit=False, extra_args=[], extra_env=None) ... [Done] ``` Pull Request resolved: https://github.com/pytorch/benchmark/pull/1635 Reviewed By: aaronenyeshi Differential Revision: D45824267 Pulled By: xuzhao9 fbshipit-source-id: 19a49f1617c325168d0c3d33cfcf24aa1d5c3ab2