pytorch

24bfcd53 - [FX] Added FX prepare_for_inference for Intel CPUs (#53805)

[FX] Added FX prepare_for_inference for Intel CPUs (#53805) Summary: Part of https://github.com/pytorch/pytorch/issues/48209 Taken from the docstring: Performs a set of optimization passes to optimize a model for the purposes of inference. Specifically, the passes that are run are: 1. Conv/BN fusion 2. Dropout removal 3. MKL layout optimizations The third optimization takes a function `use_mkl_heuristic` that's used to determine whether a subgraph should be explicity run in MKL layout. I implemented 2 heuristics: 1. Does it in MKL if the subgraph is larger than 2. 2. Benchmarks each subgraph with MKL layout and without, and keeps the subgraph if it's faster. ### Batch size of 10 and multi-threaded. Results with the second heuristic are generally as strong as the "jit.freeze" version, except in `densenet` and `vgg`, where it's faster, likely due to the heuristic being better. With the first heuristic, there are some notable gaps, particularly on `inception_v3` and `alexnet`. ``` model Eager FX FX Auto jit.mkldnn ------------ --------- --------- --------- --------- - custom 0.195614 0.14686 0.15929 0.156442 6 resnet18 0.172012 0.114007 0.119678 0.12945 6 resnet50 0.486463 0.294308 0.299518 0.318121 6 densenet161 0.955309 0.893502 0.882798 1.29315 6 inception_v3 0.38454 0.307076 0.239513 0.233083 6 googlenet 0.229388 0.237486 0.170458 0.174106 6 shufflenet 0.0513613 0.0286739 0.0292908 0.0267209 6 alexnet 0.0709602 0.0768137 0.0660831 0.0650399 6 vgg16 1.053993 0.9013264 0.9360212 1.082820 6 mobilenet 0.12264 0.0970935 0.0936568 0.106314 6 mnasnet 0.0989875 0.0412083 0.0424499 0.0472336 6 resnext 0.476811 0.315428 0.314422 0.343156 6 ``` For single-threaded (still running...) ``` model eager FX FX auto mkl threads ------------ --------- --------- --------- --------- --------- custom 0.0401415 0.259863 0.0263152 0.200667 1 resnet18 0.499931 0.382113 0.383711 0.396335 1 resnet50 1.10353 0.911865 0.923645 0.992125 1 densenet161 2.20158 2.39421 2.08204 2.30124 1 inception_v3 0.79161 0.849207 0.703546 0.724492 1 googlenet 0.66896 0.820965 0.515927 0.529414 1 shufflenet 0.0987308 0.0689343 0.0629298 0.0617193 1 alexnet 0.198795 0.19862 0.19325 0.211934 1 vgg16 3.744 3.2499 3.28503 3.31576 1 mobilenet 0.152725 0.14505 0.135555 0.159754 1 mnasnet 0.141983 0.089406 0.089599 0.0956167 1 resnext 1.13778 0.97016 0.955417 0.965376 1 ``` Pull Request resolved: https://github.com/pytorch/pytorch/pull/53805 Reviewed By: gmagogsfm Differential Revision: D27424611 Pulled By: Chillee fbshipit-source-id: a39137159de962fba7ca15121dfa9e78c1e01223