pytorch

19943aaf - [caffe2] Speed up remote net loading

[caffe2] Speed up remote net loading Summary: Training recovery takes over 3 hours for DI models. See T88118480 for more details. One of the slowness reasons could be the linear search in the ApplicationSpecificInfo. To improve that, we cache the app info into a dict so the lookup can be much faster. Test Plan: Unit test buck test caffe2/caffe2/fb/predictor:predictor_py_dist_utils_test ```Building: finished in 6.2 sec (100%) 11023/11023 jobs, 2 updated Total time: 6.6 sec More details at https://www.internalfb.com/intern/buck/build/95555464-b15f-44f2-a781-a712126aeaa1 Tpx test run coordinator for Facebook. See https://fburl.com/tpx for details. Running with tpx session id: 3f4e4913-5802-4437-81bf-1e0a08c067da Trace available for this run at /tmp/tpx-20210420-101444.394595/trace.log Started reporting to test run: https://www.internalfb.com/intern/testinfra/testrun/5348024608951863 ✓ ListingSuccess: caffe2/caffe2/fb/predictor:predictor_py_dist_utils_test - main (8.412) ✓ Pass: caffe2/caffe2/fb/predictor:predictor_py_dist_utils_test - test_empty_remote_net_in_app_into (caffe2.caffe2.fb.predictor.predictor_py_dist_utils_test.TestPredictorDistUtils) (7.844) ✓ Pass: caffe2/caffe2/fb/predictor:predictor_py_dist_utils_test - test_distributed_context_in_app_info (caffe2.caffe2.fb.predictor.predictor_py_dist_utils_test.TestPredictorDistUtils) (8.014) ✓ Pass: caffe2/caffe2/fb/predictor:predictor_py_dist_utils_test - test_remote_net_in_app_info (caffe2.caffe2.fb.predictor.predictor_py_dist_utils_test.TestPredictorDistUtils) (8.027) Summary Pass: 3 ListingSuccess: 1 If you need help debugging your runs, please follow the wiki: https://fburl.com/posting_in_tpx_users Finished test run: https://www.internalfb.com/intern/testinfra/testrun/5348024608951863 ``` Performance Test: N557020 is the old way, which takes about 30~60 secs for every 1000 remote nets N556897 is the new way, which takes 0.12 secs for every 1000 remote nets N557020 output: ~~~ I0420 112047.755 <ipython-input-2-515f8ba1b5f6>:48] Start retrieving remote nets ... I0420 112050.036 <ipython-input-2-515f8ba1b5f6>:27] Get 1000 remote nets I0420 112052.750 <ipython-input-2-515f8ba1b5f6>:27] Get 2000 remote nets I0420 112055.907 <ipython-input-2-515f8ba1b5f6>:27] Get 3000 remote nets I0420 112059.542 <ipython-input-2-515f8ba1b5f6>:27] Get 4000 remote nets I0420 112103.628 <ipython-input-2-515f8ba1b5f6>:27] Get 5000 remote nets I0420 112108.309 <ipython-input-2-515f8ba1b5f6>:27] Get 6000 remote nets I0420 112113.883 <ipython-input-2-515f8ba1b5f6>:27] Get 7000 remote nets I0420 112119.564 <ipython-input-2-515f8ba1b5f6>:27] Get 8000 remote nets I0420 112125.629 <ipython-input-2-515f8ba1b5f6>:27] Get 9000 remote nets I0420 112132.057 <ipython-input-2-515f8ba1b5f6>:27] Get 10000 remote nets I0420 112138.979 <ipython-input-2-515f8ba1b5f6>:27] Get 11000 remote nets I0420 112146.198 <ipython-input-2-515f8ba1b5f6>:27] Get 12000 remote nets I0420 112154.381 <ipython-input-2-515f8ba1b5f6>:27] Get 13000 remote nets I0420 112202.881 <ipython-input-2-515f8ba1b5f6>:27] Get 14000 remote nets I0420 112211.595 <ipython-input-2-515f8ba1b5f6>:27] Get 15000 remote nets I0420 112221.341 <ipython-input-2-515f8ba1b5f6>:27] Get 16000 remote nets I0420 112231.300 <ipython-input-2-515f8ba1b5f6>:27] Get 17000 remote nets I0420 112242.615 <ipython-input-2-515f8ba1b5f6>:27] Get 18000 remote nets I0420 112253.730 <ipython-input-2-515f8ba1b5f6>:27] Get 19000 remote nets I0420 112305.044 <ipython-input-2-515f8ba1b5f6>:27] Get 20000 remote nets I0420 112316.378 <ipython-input-2-515f8ba1b5f6>:27] Get 21000 remote nets I0420 112328.176 <ipython-input-2-515f8ba1b5f6>:27] Get 22000 remote nets I0420 112341.466 <ipython-input-2-515f8ba1b5f6>:27] Get 23000 remote nets I0420 112355.653 <ipython-input-2-515f8ba1b5f6>:27] Get 24000 remote nets I0420 112409.014 <ipython-input-2-515f8ba1b5f6>:27] Get 25000 remote nets I0420 112422.924 <ipython-input-2-515f8ba1b5f6>:27] Get 26000 remote nets I0420 112437.026 <ipython-input-2-515f8ba1b5f6>:27] Get 27000 remote nets I0420 112451.413 <ipython-input-2-515f8ba1b5f6>:27] Get 28000 remote nets I0420 112506.773 <ipython-input-2-515f8ba1b5f6>:27] Get 29000 remote nets I0420 112522.614 <ipython-input-2-515f8ba1b5f6>:27] Get 30000 remote nets I0420 112538.564 <ipython-input-2-515f8ba1b5f6>:27] Get 31000 remote nets I0420 112555.075 <ipython-input-2-515f8ba1b5f6>:27] Get 32000 remote nets I0420 112612.159 <ipython-input-2-515f8ba1b5f6>:27] Get 33000 remote nets I0420 112629.656 <ipython-input-2-515f8ba1b5f6>:27] Get 34000 remote nets I0420 112647.850 <ipython-input-2-515f8ba1b5f6>:27] Get 35000 remote nets I0420 112705.807 <ipython-input-2-515f8ba1b5f6>:27] Get 36000 remote nets I0420 112724.495 <ipython-input-2-515f8ba1b5f6>:27] Get 37000 remote nets I0420 112744.072 <ipython-input-2-515f8ba1b5f6>:27] Get 38000 remote nets I0420 112804.266 <ipython-input-2-515f8ba1b5f6>:27] Get 39000 remote nets I0420 112824.954 <ipython-input-2-515f8ba1b5f6>:27] Get 40000 remote nets I0420 112845.934 <ipython-input-2-515f8ba1b5f6>:27] Get 41000 remote nets I0420 112908.721 <ipython-input-2-515f8ba1b5f6>:27] Get 42000 remote nets I0420 112930.573 <ipython-input-2-515f8ba1b5f6>:27] Get 43000 remote nets I0420 112952.775 <ipython-input-2-515f8ba1b5f6>:27] Get 44000 remote nets I0420 113015.969 <ipython-input-2-515f8ba1b5f6>:27] Get 45000 remote nets I0420 113041.214 <ipython-input-2-515f8ba1b5f6>:27] Get 46000 remote nets I0420 113104.702 <ipython-input-2-515f8ba1b5f6>:27] Get 47000 remote nets I0420 113128.730 <ipython-input-2-515f8ba1b5f6>:27] Get 48000 remote nets I0420 113153.378 <ipython-input-2-515f8ba1b5f6>:27] Get 49000 remote nets I0420 113218.021 <ipython-input-2-515f8ba1b5f6>:27] Get 50000 remote nets I0420 113243.351 <ipython-input-2-515f8ba1b5f6>:27] Get 51000 remote nets I0420 113309.279 <ipython-input-2-515f8ba1b5f6>:27] Get 52000 remote nets I0420 113335.202 <ipython-input-2-515f8ba1b5f6>:27] Get 53000 remote nets I0420 113402.367 <ipython-input-2-515f8ba1b5f6>:27] Get 54000 remote nets I0420 113430.947 <ipython-input-2-515f8ba1b5f6>:27] Get 55000 remote nets I0420 113458.127 <ipython-input-2-515f8ba1b5f6>:27] Get 56000 remote nets I0420 113526.365 <ipython-input-2-515f8ba1b5f6>:27] Get 57000 remote nets I0420 113554.709 <ipython-input-2-515f8ba1b5f6>:27] Get 58000 remote nets I0420 113623.601 <ipython-input-2-515f8ba1b5f6>:27] Get 59000 remote nets I0420 113653.264 <ipython-input-2-515f8ba1b5f6>:27] Get 60000 remote nets I0420 113724.726 <ipython-input-2-515f8ba1b5f6>:27] Get 61000 remote nets I0420 113755.080 <ipython-input-2-515f8ba1b5f6>:27] Get 62000 remote nets I0420 113827.936 <ipython-input-2-515f8ba1b5f6>:27] Get 63000 remote nets I0420 113859.362 <ipython-input-2-515f8ba1b5f6>:27] Get 64000 remote nets I0420 113931.138 <ipython-input-2-515f8ba1b5f6>:27] Get 65000 remote nets I0420 114003.229 <ipython-input-2-515f8ba1b5f6>:27] Get 66000 remote nets I0420 114038.085 <ipython-input-2-515f8ba1b5f6>:27] Get 67000 remote nets I0420 114111.300 <ipython-input-2-515f8ba1b5f6>:27] Get 68000 remote nets I0420 114145.383 <ipython-input-2-515f8ba1b5f6>:27] Get 69000 remote nets I0420 114219.571 <ipython-input-2-515f8ba1b5f6>:27] Get 70000 remote nets I0420 114254.233 <ipython-input-2-515f8ba1b5f6>:27] Get 71000 remote nets I0420 114329.326 <ipython-input-2-515f8ba1b5f6>:27] Get 72000 remote nets I0420 114405.087 <ipython-input-2-515f8ba1b5f6>:27] Get 73000 remote nets I0420 114440.979 <ipython-input-2-515f8ba1b5f6>:27] Get 74000 remote nets I0420 114518.520 <ipython-input-2-515f8ba1b5f6>:27] Get 75000 remote nets I0420 114556.013 <ipython-input-2-515f8ba1b5f6>:27] Get 76000 remote nets I0420 114633.434 <ipython-input-2-515f8ba1b5f6>:27] Get 77000 remote nets I0420 114711.834 <ipython-input-2-515f8ba1b5f6>:27] Get 78000 remote nets I0420 114750.741 <ipython-input-2-515f8ba1b5f6>:27] Get 79000 remote nets I0420 114829.749 <ipython-input-2-515f8ba1b5f6>:27] Get 80000 remote nets I0420 114909.038 <ipython-input-2-515f8ba1b5f6>:27] Get 81000 remote nets I0420 114948.711 <ipython-input-2-515f8ba1b5f6>:27] Get 82000 remote nets I0420 115028.869 <ipython-input-2-515f8ba1b5f6>:27] Get 83000 remote nets I0420 115109.094 <ipython-input-2-515f8ba1b5f6>:27] Get 84000 remote nets I0420 115150.249 <ipython-input-2-515f8ba1b5f6>:27] Get 85000 remote nets I0420 115231.601 <ipython-input-2-515f8ba1b5f6>:27] Get 86000 remote nets I0420 115313.772 <ipython-input-2-515f8ba1b5f6>:27] Get 87000 remote nets I0420 115356.035 <ipython-input-2-515f8ba1b5f6>:27] Get 88000 remote nets I0420 115438.846 <ipython-input-2-515f8ba1b5f6>:27] Get 89000 remote nets I0420 115522.213 <ipython-input-2-515f8ba1b5f6>:27] Get 90000 remote nets I0420 115607.908 <ipython-input-2-515f8ba1b5f6>:27] Get 91000 remote nets I0420 115652.009 <ipython-input-2-515f8ba1b5f6>:27] Get 92000 remote nets I0420 115736.510 <ipython-input-2-515f8ba1b5f6>:27] Get 93000 remote nets I0420 115822.303 <ipython-input-2-515f8ba1b5f6>:27] Get 94000 remote nets I0420 115908.392 <ipython-input-2-515f8ba1b5f6>:27] Get 95000 remote nets I0420 115954.912 <ipython-input-2-515f8ba1b5f6>:27] Get 96000 remote nets I0420 120042.219 <ipython-input-2-515f8ba1b5f6>:27] Get 97000 remote nets I0420 120129.969 <ipython-input-2-515f8ba1b5f6>:27] Get 98000 remote nets I0420 120218.765 <ipython-input-2-515f8ba1b5f6>:27] Get 99000 remote nets I0420 120306.883 <ipython-input-2-515f8ba1b5f6>:27] Get 100000 remote nets I0420 120355.543 <ipython-input-2-515f8ba1b5f6>:27] Get 101000 remote nets I0420 120444.976 <ipython-input-2-515f8ba1b5f6>:27] Get 102000 remote nets I0420 120533.482 <ipython-input-2-515f8ba1b5f6>:27] Get 103000 remote nets I0420 120622.351 <ipython-input-2-515f8ba1b5f6>:27] Get 104000 remote nets I0420 120712.467 <ipython-input-2-515f8ba1b5f6>:27] Get 105000 remote nets I0420 120802.660 <ipython-input-2-515f8ba1b5f6>:27] Get 106000 remote nets I0420 120854.634 <ipython-input-2-515f8ba1b5f6>:27] Get 107000 remote nets I0420 120945.786 <ipython-input-2-515f8ba1b5f6>:27] Get 108000 remote nets ~~~ N556897 output: ~~~ I0420 111502.516 <ipython-input-7-52640a51556f>:60] Start retrieving remote nets ... I0420 111504.709 <ipython-input-7-52640a51556f>:40] Get 1000 remote nets I0420 111504.825 <ipython-input-7-52640a51556f>:40] Get 2000 remote nets I0420 111504.941 <ipython-input-7-52640a51556f>:40] Get 3000 remote nets I0420 111505.056 <ipython-input-7-52640a51556f>:40] Get 4000 remote nets I0420 111505.174 <ipython-input-7-52640a51556f>:40] Get 5000 remote nets I0420 111505.286 <ipython-input-7-52640a51556f>:40] Get 6000 remote nets I0420 111505.405 <ipython-input-7-52640a51556f>:40] Get 7000 remote nets I0420 111505.522 <ipython-input-7-52640a51556f>:40] Get 8000 remote nets I0420 111505.639 <ipython-input-7-52640a51556f>:40] Get 9000 remote nets I0420 111505.756 <ipython-input-7-52640a51556f>:40] Get 10000 remote nets I0420 111505.873 <ipython-input-7-52640a51556f>:40] Get 11000 remote nets I0420 111505.990 <ipython-input-7-52640a51556f>:40] Get 12000 remote nets I0420 111506.106 <ipython-input-7-52640a51556f>:40] Get 13000 remote nets I0420 111506.223 <ipython-input-7-52640a51556f>:40] Get 14000 remote nets I0420 111506.343 <ipython-input-7-52640a51556f>:40] Get 15000 remote nets I0420 111506.457 <ipython-input-7-52640a51556f>:40] Get 16000 remote nets I0420 111506.585 <ipython-input-7-52640a51556f>:40] Get 17000 remote nets I0420 111508.930 <ipython-input-7-52640a51556f>:40] Get 18000 remote nets I0420 111509.045 <ipython-input-7-52640a51556f>:40] Get 19000 remote nets I0420 111509.154 <ipython-input-7-52640a51556f>:40] Get 20000 remote nets I0420 111509.266 <ipython-input-7-52640a51556f>:40] Get 21000 remote nets I0420 111509.382 <ipython-input-7-52640a51556f>:40] Get 22000 remote nets I0420 111509.497 <ipython-input-7-52640a51556f>:40] Get 23000 remote nets I0420 111509.614 <ipython-input-7-52640a51556f>:40] Get 24000 remote nets I0420 111509.736 <ipython-input-7-52640a51556f>:40] Get 25000 remote nets I0420 111509.854 <ipython-input-7-52640a51556f>:40] Get 26000 remote nets I0420 111509.972 <ipython-input-7-52640a51556f>:40] Get 27000 remote nets I0420 111510.090 <ipython-input-7-52640a51556f>:40] Get 28000 remote nets I0420 111510.210 <ipython-input-7-52640a51556f>:40] Get 29000 remote nets I0420 111510.329 <ipython-input-7-52640a51556f>:40] Get 30000 remote nets I0420 111510.448 <ipython-input-7-52640a51556f>:40] Get 31000 remote nets I0420 111510.572 <ipython-input-7-52640a51556f>:40] Get 32000 remote nets I0420 111510.689 <ipython-input-7-52640a51556f>:40] Get 33000 remote nets I0420 111510.821 <ipython-input-7-52640a51556f>:40] Get 34000 remote nets I0420 111510.989 <ipython-input-7-52640a51556f>:40] Get 35000 remote nets I0420 111511.110 <ipython-input-7-52640a51556f>:40] Get 36000 remote nets I0420 111511.236 <ipython-input-7-52640a51556f>:40] Get 37000 remote nets I0420 111511.357 <ipython-input-7-52640a51556f>:40] Get 38000 remote nets I0420 111511.482 <ipython-input-7-52640a51556f>:40] Get 39000 remote nets I0420 111511.607 <ipython-input-7-52640a51556f>:40] Get 40000 remote nets I0420 111511.729 <ipython-input-7-52640a51556f>:40] Get 41000 remote nets I0420 111511.855 <ipython-input-7-52640a51556f>:40] Get 42000 remote nets I0420 111511.988 <ipython-input-7-52640a51556f>:40] Get 43000 remote nets I0420 111512.112 <ipython-input-7-52640a51556f>:40] Get 44000 remote nets I0420 111512.232 <ipython-input-7-52640a51556f>:40] Get 45000 remote nets I0420 111512.353 <ipython-input-7-52640a51556f>:40] Get 46000 remote nets I0420 111512.477 <ipython-input-7-52640a51556f>:40] Get 47000 remote nets I0420 111512.597 <ipython-input-7-52640a51556f>:40] Get 48000 remote nets I0420 111512.723 <ipython-input-7-52640a51556f>:40] Get 49000 remote nets I0420 111512.839 <ipython-input-7-52640a51556f>:40] Get 50000 remote nets I0420 111512.969 <ipython-input-7-52640a51556f>:40] Get 51000 remote nets I0420 111513.085 <ipython-input-7-52640a51556f>:40] Get 52000 remote nets I0420 111513.205 <ipython-input-7-52640a51556f>:40] Get 53000 remote nets I0420 111513.322 <ipython-input-7-52640a51556f>:40] Get 54000 remote nets I0420 111513.441 <ipython-input-7-52640a51556f>:40] Get 55000 remote nets I0420 111513.559 <ipython-input-7-52640a51556f>:40] Get 56000 remote nets I0420 111513.678 <ipython-input-7-52640a51556f>:40] Get 57000 remote nets I0420 111513.796 <ipython-input-7-52640a51556f>:40] Get 58000 remote nets I0420 111513.918 <ipython-input-7-52640a51556f>:40] Get 59000 remote nets I0420 111514.038 <ipython-input-7-52640a51556f>:40] Get 60000 remote nets I0420 111514.158 <ipython-input-7-52640a51556f>:40] Get 61000 remote nets I0420 111514.273 <ipython-input-7-52640a51556f>:40] Get 62000 remote nets I0420 111514.391 <ipython-input-7-52640a51556f>:40] Get 63000 remote nets I0420 111514.512 <ipython-input-7-52640a51556f>:40] Get 64000 remote nets I0420 111514.638 <ipython-input-7-52640a51556f>:40] Get 65000 remote nets I0420 111514.759 <ipython-input-7-52640a51556f>:40] Get 66000 remote nets I0420 111514.874 <ipython-input-7-52640a51556f>:40] Get 67000 remote nets I0420 111515.000 <ipython-input-7-52640a51556f>:40] Get 68000 remote nets I0420 111515.117 <ipython-input-7-52640a51556f>:40] Get 69000 remote nets I0420 111515.235 <ipython-input-7-52640a51556f>:40] Get 70000 remote nets I0420 111515.358 <ipython-input-7-52640a51556f>:40] Get 71000 remote nets I0420 111515.481 <ipython-input-7-52640a51556f>:40] Get 72000 remote nets I0420 111515.604 <ipython-input-7-52640a51556f>:40] Get 73000 remote nets I0420 111515.725 <ipython-input-7-52640a51556f>:40] Get 74000 remote nets I0420 111515.848 <ipython-input-7-52640a51556f>:40] Get 75000 remote nets I0420 111515.979 <ipython-input-7-52640a51556f>:40] Get 76000 remote nets I0420 111516.102 <ipython-input-7-52640a51556f>:40] Get 77000 remote nets I0420 111516.226 <ipython-input-7-52640a51556f>:40] Get 78000 remote nets I0420 111516.344 <ipython-input-7-52640a51556f>:40] Get 79000 remote nets I0420 111516.472 <ipython-input-7-52640a51556f>:40] Get 80000 remote nets I0420 111516.603 <ipython-input-7-52640a51556f>:40] Get 81000 remote nets I0420 111516.751 <ipython-input-7-52640a51556f>:40] Get 82000 remote nets I0420 111516.883 <ipython-input-7-52640a51556f>:40] Get 83000 remote nets I0420 111517.025 <ipython-input-7-52640a51556f>:40] Get 84000 remote nets I0420 111517.160 <ipython-input-7-52640a51556f>:40] Get 85000 remote nets I0420 111517.290 <ipython-input-7-52640a51556f>:40] Get 86000 remote nets I0420 111517.415 <ipython-input-7-52640a51556f>:40] Get 87000 remote nets I0420 111517.541 <ipython-input-7-52640a51556f>:40] Get 88000 remote nets I0420 111517.665 <ipython-input-7-52640a51556f>:40] Get 89000 remote nets I0420 111517.790 <ipython-input-7-52640a51556f>:40] Get 90000 remote nets I0420 111517.918 <ipython-input-7-52640a51556f>:40] Get 91000 remote nets I0420 111518.044 <ipython-input-7-52640a51556f>:40] Get 92000 remote nets I0420 111518.171 <ipython-input-7-52640a51556f>:40] Get 93000 remote nets I0420 111518.292 <ipython-input-7-52640a51556f>:40] Get 94000 remote nets I0420 111518.429 <ipython-input-7-52640a51556f>:40] Get 95000 remote nets I0420 111520.024 <ipython-input-7-52640a51556f>:40] Get 96000 remote nets I0420 111520.148 <ipython-input-7-52640a51556f>:40] Get 97000 remote nets I0420 111520.271 <ipython-input-7-52640a51556f>:40] Get 98000 remote nets I0420 111520.396 <ipython-input-7-52640a51556f>:40] Get 99000 remote nets I0420 111520.522 <ipython-input-7-52640a51556f>:40] Get 100000 remote nets I0420 111520.646 <ipython-input-7-52640a51556f>:40] Get 101000 remote nets I0420 111520.770 <ipython-input-7-52640a51556f>:40] Get 102000 remote nets I0420 111520.899 <ipython-input-7-52640a51556f>:40] Get 103000 remote nets I0420 111521.023 <ipython-input-7-52640a51556f>:40] Get 104000 remote nets I0420 111521.149 <ipython-input-7-52640a51556f>:40] Get 105000 remote nets I0420 111521.274 <ipython-input-7-52640a51556f>:40] Get 106000 remote nets I0420 111521.399 <ipython-input-7-52640a51556f>:40] Get 107000 remote nets I0420 111521.526 <ipython-input-7-52640a51556f>:40] Get 108000 remote nets I0420 111521.651 <ipython-input-7-52640a51556f>:40] Get 109000 remote nets I0420 111521.778 <ipython-input-7-52640a51556f>:40] Get 110000 remote nets I0420 111521.900 <ipython-input-7-52640a51556f>:40] Get 111000 remote nets I0420 111522.055 <ipython-input-7-52640a51556f>:40] Get 112000 remote nets I0420 111522.173 <ipython-input-7-52640a51556f>:40] Get 113000 remote nets I0420 111522.297 <ipython-input-7-52640a51556f>:40] Get 114000 remote nets I0420 111522.421 <ipython-input-7-52640a51556f>:40] Get 115000 remote nets I0420 111522.545 <ipython-input-7-52640a51556f>:40] Get 116000 remote nets I0420 111522.671 <ipython-input-7-52640a51556f>:40] Get 117000 remote nets I0420 111522.795 <ipython-input-7-52640a51556f>:40] Get 118000 remote nets I0420 111522.919 <ipython-input-7-52640a51556f>:40] Get 119000 remote nets I0420 111523.048 <ipython-input-7-52640a51556f>:40] Get 120000 remote nets I0420 111523.171 <ipython-input-7-52640a51556f>:40] Get 121000 remote nets I0420 111523.298 <ipython-input-7-52640a51556f>:40] Get 122000 remote nets I0420 111523.420 <ipython-input-7-52640a51556f>:40] Get 123000 remote nets I0420 111523.544 <ipython-input-7-52640a51556f>:40] Get 124000 remote nets I0420 111523.669 <ipython-input-7-52640a51556f>:40] Get 125000 remote nets I0420 111523.794 <ipython-input-7-52640a51556f>:40] Get 126000 remote nets I0420 111523.920 <ipython-input-7-52640a51556f>:40] Get 127000 remote nets I0420 111524.041 <ipython-input-7-52640a51556f>:40] Get 128000 remote nets I0420 111524.173 <ipython-input-7-52640a51556f>:40] Get 129000 remote nets I0420 111524.293 <ipython-input-7-52640a51556f>:40] Get 130000 remote nets I0420 111524.417 <ipython-input-7-52640a51556f>:40] Get 131000 remote nets I0420 111524.542 <ipython-input-7-52640a51556f>:40] Get 132000 remote nets I0420 111524.665 <ipython-input-7-52640a51556f>:40] Get 133000 remote nets I0420 111524.790 <ipython-input-7-52640a51556f>:40] Get 134000 remote nets I0420 111524.913 <ipython-input-7-52640a51556f>:40] Get 135000 remote nets I0420 111525.038 <ipython-input-7-52640a51556f>:40] Get 136000 remote nets I0420 111525.166 <ipython-input-7-52640a51556f>:40] Get 137000 remote nets I0420 111525.289 <ipython-input-7-52640a51556f>:40] Get 138000 remote nets I0420 111525.414 <ipython-input-7-52640a51556f>:40] Get 139000 remote nets I0420 111525.536 <ipython-input-7-52640a51556f>:40] Get 140000 remote nets I0420 111525.659 <ipython-input-7-52640a51556f>:40] Get 141000 remote nets I0420 111525.782 <ipython-input-7-52640a51556f>:40] Get 142000 remote nets I0420 111525.907 <ipython-input-7-52640a51556f>:40] Get 143000 remote nets I0420 111526.035 <ipython-input-7-52640a51556f>:40] Get 144000 remote nets I0420 111526.157 <ipython-input-7-52640a51556f>:40] Get 145000 remote nets I0420 111526.287 <ipython-input-7-52640a51556f>:40] Get 146000 remote nets I0420 111526.409 <ipython-input-7-52640a51556f>:40] Get 147000 remote nets I0420 111526.533 <ipython-input-7-52640a51556f>:40] Get 148000 remote nets I0420 111526.658 <ipython-input-7-52640a51556f>:40] Get 149000 remote nets I0420 111526.781 <ipython-input-7-52640a51556f>:40] Get 150000 remote nets I0420 111526.908 <ipython-input-7-52640a51556f>:40] Get 151000 remote nets I0420 111527.033 <ipython-input-7-52640a51556f>:40] Get 152000 remote nets I0420 111527.158 <ipython-input-7-52640a51556f>:40] Get 153000 remote nets I0420 111527.289 <ipython-input-7-52640a51556f>:40] Get 154000 remote nets I0420 111527.413 <ipython-input-7-52640a51556f>:40] Get 155000 remote nets I0420 111527.544 <ipython-input-7-52640a51556f>:40] Get 156000 remote nets I0420 111527.665 <ipython-input-7-52640a51556f>:40] Get 157000 remote nets I0420 111527.790 <ipython-input-7-52640a51556f>:40] Get 158000 remote nets I0420 111527.917 <ipython-input-7-52640a51556f>:40] Get 159000 remote nets I0420 111528.046 <ipython-input-7-52640a51556f>:40] Get 160000 remote nets I0420 111528.175 <ipython-input-7-52640a51556f>:40] Get 161000 remote nets I0420 111528.297 <ipython-input-7-52640a51556f>:40] Get 162000 remote nets I0420 111528.422 <ipython-input-7-52640a51556f>:40] Get 163000 remote nets I0420 111528.548 <ipython-input-7-52640a51556f>:40] Get 164000 remote nets I0420 111528.672 <ipython-input-7-52640a51556f>:40] Get 165000 remote nets I0420 111528.796 <ipython-input-7-52640a51556f>:40] Get 166000 remote nets I0420 111528.920 <ipython-input-7-52640a51556f>:40] Get 167000 remote nets I0420 111529.045 <ipython-input-7-52640a51556f>:40] Get 168000 remote nets I0420 111529.172 <ipython-input-7-52640a51556f>:40] Get 169000 remote nets I0420 111529.300 <ipython-input-7-52640a51556f>:40] Get 170000 remote nets I0420 111529.426 <ipython-input-7-52640a51556f>:40] Get 171000 remote nets I0420 111529.547 <ipython-input-7-52640a51556f>:40] Get 172000 remote nets I0420 111529.683 <ipython-input-7-52640a51556f>:40] Get 173000 remote nets I0420 111529.800 <ipython-input-7-52640a51556f>:40] Get 174000 remote nets I0420 111529.923 <ipython-input-7-52640a51556f>:40] Get 175000 remote nets I0420 111530.080 <ipython-input-7-52640a51556f>:40] Get 176000 remote nets I0420 111530.205 <ipython-input-7-52640a51556f>:40] Get 177000 remote nets I0420 111530.331 <ipython-input-7-52640a51556f>:40] Get 178000 remote nets I0420 111530.453 <ipython-input-7-52640a51556f>:40] Get 179000 remote nets I0420 111530.577 <ipython-input-7-52640a51556f>:40] Get 180000 remote nets I0420 111530.705 <ipython-input-7-52640a51556f>:40] Get 181000 remote nets I0420 111530.829 <ipython-input-7-52640a51556f>:40] Get 182000 remote nets I0420 111530.955 <ipython-input-7-52640a51556f>:40] Get 183000 remote nets I0420 111531.082 <ipython-input-7-52640a51556f>:40] Get 184000 remote nets I0420 111531.210 <ipython-input-7-52640a51556f>:40] Get 185000 remote nets I0420 111531.338 <ipython-input-7-52640a51556f>:40] Get 186000 remote nets I0420 111531.461 <ipython-input-7-52640a51556f>:40] Get 187000 remote nets I0420 111531.588 <ipython-input-7-52640a51556f>:40] Get 188000 remote nets I0420 111531.708 <ipython-input-7-52640a51556f>:40] Get 189000 remote nets I0420 111531.845 <ipython-input-7-52640a51556f>:40] Get 190000 remote nets I0420 111531.968 <ipython-input-7-52640a51556f>:40] Get 191000 remote nets I0420 111532.096 <ipython-input-7-52640a51556f>:40] Get 192000 remote nets I0420 111534.047 <ipython-input-7-52640a51556f>:40] Get 193000 remote nets I0420 111534.172 <ipython-input-7-52640a51556f>:40] Get 194000 remote nets I0420 111534.297 <ipython-input-7-52640a51556f>:40] Get 195000 remote nets I0420 111534.420 <ipython-input-7-52640a51556f>:40] Get 196000 remote nets I0420 111534.543 <ipython-input-7-52640a51556f>:40] Get 197000 remote nets I0420 111534.671 <ipython-input-7-52640a51556f>:40] Get 198000 remote nets I0420 111534.794 <ipython-input-7-52640a51556f>:40] Get 199000 remote nets I0420 111534.920 <ipython-input-7-52640a51556f>:40] Get 200000 remote nets I0420 111535.044 <ipython-input-7-52640a51556f>:40] Get 201000 remote nets I0420 111535.167 <ipython-input-7-52640a51556f>:40] Get 202000 remote nets I0420 111535.291 <ipython-input-7-52640a51556f>:40] Get 203000 remote nets I0420 111537.169 <ipython-input-7-52640a51556f>:64] Finish retrieving remote nets. Starting processing ... I0420 111537.201 <ipython-input-7-52640a51556f>:77] Finished processing remote nets ~~~ Reviewed By: heslami Differential Revision: D27886217 fbshipit-source-id: cdc398d04bf963d4f495adc0a91c8ceb54466e58