pytorch

8a6173ed - [caffe2] tune prefetch distance

[caffe2] tune prefetch distance Summary: As title Test Plan: Seeing prefetch distance 8 is slightly faster overall. Probably because prefetch distance 16 was tuned for int8 but int4 is a bit slower. Comparing 4, 8, and 16 SKL T6 ``` bit_rate 4batch size 1 num rows 5000000 emb dim 112 avg length 100 | bit_rate 4batch size 1 num rows 5000000 emb dim 112 avg length 100 | bit_rate 4batch size 1 num rows 5000000 emb dim 112 avg length 100 64 bit indices with prefetching, lengths_sum 100 | 64 bit indices with prefetching, lengths_sum 100 | 64 bit indices with prefetching, lengths_sum 100 SLS cache not flushed prefetch 4 b/w 3.23747 GB/s effective b/w: 3.4533GB/s | SLS cache not flushed prefetch 8 b/w 3.14268 GB/s effective b/w: 3.35219GB/s | SLS cache not flushed prefetch 16 b/w 3.31473 GB/s effective b/w: 3.53572GB/s SLS cache flushed prefetch 4 b/w 0.953971 GB/s effective b/w: 1.01757GB/s | SLS cache flushed prefetch 8 b/w 0.908884 GB/s effective b/w: 0.969477GB/s | SLS cache flushed prefetch 16 b/w 0.858664 GB/s effective b/w: 0.915908GB/s bit_rate 4batch size 1 num rows 6000000 emb dim 24 avg length 100 | bit_rate 4batch size 1 num rows 6000000 emb dim 24 avg length 100 | bit_rate 4batch size 1 num rows 6000000 emb dim 24 avg length 100 64 bit indices with prefetching, lengths_sum 100 | 64 bit indices with prefetching, lengths_sum 100 | 64 bit indices with prefetching, lengths_sum 100 SLS cache not flushed prefetch 4 b/w 0.738689 GB/s effective b/w: 2.95476GB/s | SLS cache not flushed prefetch 8 b/w 0.593186 GB/s effective b/w: 2.37274GB/s | SLS cache not flushed prefetch 16 b/w 0.654879 GB/s effective b/w: 2.61952GB/s SLS cache flushed prefetch 4 b/w 0.422531 GB/s effective b/w: 1.69013GB/s | SLS cache flushed prefetch 8 b/w 0.525107 GB/s effective b/w: 2.10043GB/s | SLS cache flushed prefetch 16 b/w 0.509311 GB/s effective b/w: 2.03724GB/s bit_rate 4batch size 1 num rows 2000000 emb dim 20 avg length 50 | bit_rate 4batch size 1 num rows 2000000 emb dim 20 avg length 50 | bit_rate 4batch size 1 num rows 2000000 emb dim 20 avg length 50 64 bit indices with prefetching, lengths_sum 50 | 64 bit indices with prefetching, lengths_sum 50 | 64 bit indices with prefetching, lengths_sum 50 SLS cache not flushed prefetch 4 b/w 0.28347 GB/s effective b/w: 1.29586GB/s | SLS cache not flushed prefetch 8 b/w 0.273341 GB/s effective b/w: 1.24956GB/s | SLS cache not flushed prefetch 16 b/w 0.550877 GB/s effective b/w: 2.5183GB/s SLS cache flushed prefetch 4 b/w 0.348034 GB/s effective b/w: 1.59101GB/s | SLS cache flushed prefetch 8 b/w 0.359454 GB/s effective b/w: 1.64322GB/s | SLS cache flushed prefetch 16 b/w 0.348086 GB/s effective b/w: 1.59125GB/s bit_rate 4batch size 1 num rows 6000000 emb dim 24 avg length 50 | bit_rate 4batch size 1 num rows 6000000 emb dim 24 avg length 50 | bit_rate 4batch size 1 num rows 6000000 emb dim 24 avg length 50 64 bit indices with prefetching, lengths_sum 50 | 64 bit indices with prefetching, lengths_sum 50 | 64 bit indices with prefetching, lengths_sum 50 SLS cache not flushed prefetch 4 b/w 0.322958 GB/s effective b/w: 1.29183GB/s | SLS cache not flushed prefetch 8 b/w 0.622423 GB/s effective b/w: 2.48969GB/s | SLS cache not flushed prefetch 16 b/w 0.331167 GB/s effective b/w: 1.32467GB/s SLS cache flushed prefetch 4 b/w 0.406938 GB/s effective b/w: 1.62775GB/s | SLS cache flushed prefetch 8 b/w 0.548998 GB/s effective b/w: 2.19599GB/s | SLS cache flushed prefetch 16 b/w 0.40833 GB/s effective b/w: 1.63332GB/s bit_rate 4batch size 1 num rows 4000000 emb dim 68 avg length 100 | bit_rate 4batch size 1 num rows 4000000 emb dim 68 avg length 100 | bit_rate 4batch size 1 num rows 4000000 emb dim 68 avg length 100 64 bit indices with prefetching, lengths_sum 100 | 64 bit indices with prefetching, lengths_sum 100 | 64 bit indices with prefetching, lengths_sum 100 SLS cache not flushed prefetch 4 b/w 2.26016 GB/s effective b/w: 3.80658GB/s | SLS cache not flushed prefetch 8 b/w 1.66055 GB/s effective b/w: 2.79671GB/s | SLS cache not flushed prefetch 16 b/w 2.31538 GB/s effective b/w: 3.89959GB/s SLS cache flushed prefetch 4 b/w 0.714837 GB/s effective b/w: 1.20394GB/s | SLS cache flushed prefetch 8 b/w 0.659482 GB/s effective b/w: 1.11071GB/s | SLS cache flushed prefetch 16 b/w 0.643239 GB/s effective b/w: 1.08335GB/s bit_rate 4batch size 16 num rows 10000000 emb dim 28 avg length 100 | bit_rate 4batch size 16 num rows 10000000 emb dim 28 avg length 100 | bit_rate 4batch size 16 num rows 10000000 emb dim 28 avg length 100 64 bit indices with prefetching, lengths_sum 1600 | 64 bit indices with prefetching, lengths_sum 1600 | 64 bit indices with prefetching, lengths_sum 1600 SLS cache not flushed prefetch 4 b/w 2.33704 GB/s effective b/w: 8.30946GB/s | SLS cache not flushed prefetch 8 b/w 2.53271 GB/s effective b/w: 9.00521GB/s | SLS cache not flushed prefetch 16 b/w 2.27881 GB/s effective b/w: 8.10242GB/s SLS cache flushed prefetch 4 b/w 0.594799 GB/s effective b/w: 2.11484GB/s | SLS cache flushed prefetch 8 b/w 0.675113 GB/s effective b/w: 2.4004GB/s | SLS cache flushed prefetch 16 b/w 0.681539 GB/s effective b/w: 2.42325GB/s bit_rate 4batch size 1 num rows 2000000 emb dim 20 avg length 100 | bit_rate 4batch size 1 num rows 2000000 emb dim 20 avg length 100 | bit_rate 4batch size 1 num rows 2000000 emb dim 20 avg length 100 64 bit indices with prefetching, lengths_sum 100 | 64 bit indices with prefetching, lengths_sum 100 | 64 bit indices with prefetching, lengths_sum 100 SLS cache not flushed prefetch 4 b/w 0.564903 GB/s effective b/w: 2.58242GB/s | SLS cache not flushed prefetch 8 b/w 0.416964 GB/s effective b/w: 1.90612GB/s | SLS cache not flushed prefetch 16 b/w 0.544874 GB/s effective b/w: 2.49085GB/s SLS cache flushed prefetch 4 b/w 0.342759 GB/s effective b/w: 1.5669GB/s | SLS cache flushed prefetch 8 b/w 0.394844 GB/s effective b/w: 1.805GB/s | SLS cache flushed prefetch 16 b/w 0.339888 GB/s effective b/w: 1.55378GB/s bit_rate 4batch size 40 num rows 5000000 emb dim 12 avg length 40 | bit_rate 4batch size 40 num rows 5000000 emb dim 12 avg length 40 | bit_rate 4batch size 40 num rows 5000000 emb dim 12 avg length 40 64 bit indices with prefetching, lengths_sum 1600 | 64 bit indices with prefetching, lengths_sum 1600 | 64 bit indices with prefetching, lengths_sum 1600 SLS cache not flushed prefetch 4 b/w 1.72766 GB/s effective b/w: 11.057GB/s | SLS cache not flushed prefetch 8 b/w 1.77403 GB/s effective b/w: 11.3538GB/s | SLS cache not flushed prefetch 16 b/w 1.72637 GB/s effective b/w: 11.0488GB/s SLS cache flushed prefetch 4 b/w 0.388679 GB/s effective b/w: 2.48754GB/s | SLS cache flushed prefetch 8 b/w 0.440318 GB/s effective b/w: 2.81803GB/s | SLS cache flushed prefetch 16 b/w 0.46335 GB/s effective b/w: 2.96544GB/s ``` BDW T6 ``` bit_rate 4batch size 1 num rows 5000000 emb dim 112 avg length 100 | bit_rate 4batch size 1 num rows 5000000 emb dim 112 avg length 100 | bit_rate 4batch size 1 num rows 5000000 emb dim 112 avg length 100 64 bit indices with prefetching, lengths_sum 100 | 64 bit indices with prefetching, lengths_sum 100 | 64 bit indices with prefetching, lengths_sum 100 SLS cache not flushed prefetch 4 b/w 0.160859 GB/s effective b/w: 0.171583GB/s | SLS cache not flushed prefetch 8 b/w 0.153157 GB/s effective b/w: 0.163367GB/s | SLS cache not flushed prefetch 16 b/w 0.13472 GB/s effective b/w: 0.143701GB/s SLS cache flushed prefetch 4 b/w 0.147863 GB/s effective b/w: 0.157721GB/s | SLS cache flushed prefetch 8 b/w 0.127365 GB/s effective b/w: 0.135856GB/s | SLS cache flushed prefetch 16 b/w 0.147118 GB/s effective b/w: 0.156926GB/s bit_rate 4batch size 1 num rows 6000000 emb dim 24 avg length 100 | bit_rate 4batch size 1 num rows 6000000 emb dim 24 avg length 100 | bit_rate 4batch size 1 num rows 6000000 emb dim 24 avg length 100 64 bit indices with prefetching, lengths_sum 100 | 64 bit indices with prefetching, lengths_sum 100 | 64 bit indices with prefetching, lengths_sum 100 SLS cache not flushed prefetch 4 b/w 0.190173 GB/s effective b/w: 0.760691GB/s | SLS cache not flushed prefetch 8 b/w 0.275278 GB/s effective b/w: 1.10111GB/s | SLS cache not flushed prefetch 16 b/w 0.190026 GB/s effective b/w: 0.760104GB/s SLS cache flushed prefetch 4 b/w 0.160147 GB/s effective b/w: 0.640589GB/s | SLS cache flushed prefetch 8 b/w 0.188328 GB/s effective b/w: 0.753313GB/s | SLS cache flushed prefetch 16 b/w 0.168198 GB/s effective b/w: 0.672792GB/s bit_rate 4batch size 1 num rows 2000000 emb dim 20 avg length 50 | bit_rate 4batch size 1 num rows 2000000 emb dim 20 avg length 50 | bit_rate 4batch size 1 num rows 2000000 emb dim 20 avg length 50 64 bit indices with prefetching, lengths_sum 50 | 64 bit indices with prefetching, lengths_sum 50 | 64 bit indices with prefetching, lengths_sum 50 SLS cache not flushed prefetch 4 b/w 0.240228 GB/s effective b/w: 1.09818GB/s | SLS cache not flushed prefetch 8 b/w 0.239071 GB/s effective b/w: 1.0929GB/s | SLS cache not flushed prefetch 16 b/w 0.213773 GB/s effective b/w: 0.977248GB/s SLS cache flushed prefetch 4 b/w 0.144547 GB/s effective b/w: 0.660788GB/s | SLS cache flushed prefetch 8 b/w 0.143015 GB/s effective b/w: 0.653782GB/s | SLS cache flushed prefetch 16 b/w 0.143064 GB/s effective b/w: 0.654009GB/s bit_rate 4batch size 1 num rows 6000000 emb dim 24 avg length 50 | bit_rate 4batch size 1 num rows 6000000 emb dim 24 avg length 50 | bit_rate 4batch size 1 num rows 6000000 emb dim 24 avg length 50 64 bit indices with prefetching, lengths_sum 50 | 64 bit indices with prefetching, lengths_sum 50 | 64 bit indices with prefetching, lengths_sum 50 SLS cache not flushed prefetch 4 b/w 0.271859 GB/s effective b/w: 1.08744GB/s | SLS cache not flushed prefetch 8 b/w 0.236553 GB/s effective b/w: 0.946214GB/s | SLS cache not flushed prefetch 16 b/w 0.271416 GB/s effective b/w: 1.08567GB/s SLS cache flushed prefetch 4 b/w 0.175296 GB/s effective b/w: 0.701185GB/s | SLS cache flushed prefetch 8 b/w 0.172102 GB/s effective b/w: 0.688409GB/s | SLS cache flushed prefetch 16 b/w 0.167294 GB/s effective b/w: 0.669176GB/s bit_rate 4batch size 1 num rows 4000000 emb dim 68 avg length 100 | bit_rate 4batch size 1 num rows 4000000 emb dim 68 avg length 100 | bit_rate 4batch size 1 num rows 4000000 emb dim 68 avg length 100 64 bit indices with prefetching, lengths_sum 100 | 64 bit indices with prefetching, lengths_sum 100 | 64 bit indices with prefetching, lengths_sum 100 SLS cache not flushed prefetch 4 b/w 0.169403 GB/s effective b/w: 0.285311GB/s | SLS cache not flushed prefetch 8 b/w 0.180288 GB/s effective b/w: 0.303643GB/s | SLS cache not flushed prefetch 16 b/w 0.177599 GB/s effective b/w: 0.299114GB/s SLS cache flushed prefetch 4 b/w 0.126442 GB/s effective b/w: 0.212955GB/s | SLS cache flushed prefetch 8 b/w 0.134837 GB/s effective b/w: 0.227094GB/s | SLS cache flushed prefetch 16 b/w 0.130753 GB/s effective b/w: 0.220216GB/s bit_rate 4batch size 16 num rows 10000000 emb dim 28 avg length 100 | bit_rate 4batch size 16 num rows 10000000 emb dim 28 avg length 100 | bit_rate 4batch size 16 num rows 10000000 emb dim 28 avg length 100 64 bit indices with prefetching, lengths_sum 1600 | 64 bit indices with prefetching, lengths_sum 1600 | 64 bit indices with prefetching, lengths_sum 1600 SLS cache not flushed prefetch 4 b/w 0.152165 GB/s effective b/w: 0.541032GB/s | SLS cache not flushed prefetch 8 b/w 0.153995 GB/s effective b/w: 0.547538GB/s | SLS cache not flushed prefetch 16 b/w 0.157132 GB/s effective b/w: 0.558692GB/s SLS cache flushed prefetch 4 b/w 0.144367 GB/s effective b/w: 0.513307GB/s | SLS cache flushed prefetch 8 b/w 0.151356 GB/s effective b/w: 0.538156GB/s | SLS cache flushed prefetch 16 b/w 0.152418 GB/s effective b/w: 0.541932GB/s bit_rate 4batch size 1 num rows 2000000 emb dim 20 avg length 100 | bit_rate 4batch size 1 num rows 2000000 emb dim 20 avg length 100 | bit_rate 4batch size 1 num rows 2000000 emb dim 20 avg length 100 64 bit indices with prefetching, lengths_sum 100 | 64 bit indices with prefetching, lengths_sum 100 | 64 bit indices with prefetching, lengths_sum 100 SLS cache not flushed prefetch 4 b/w 0.221365 GB/s effective b/w: 1.01195GB/s | SLS cache not flushed prefetch 8 b/w 0.242639 GB/s effective b/w: 1.1092GB/s | SLS cache not flushed prefetch 16 b/w 0.24001 GB/s effective b/w: 1.09719GB/s SLS cache flushed prefetch 4 b/w 0.141794 GB/s effective b/w: 0.6482GB/s | SLS cache flushed prefetch 8 b/w 0.13777 GB/s effective b/w: 0.629803GB/s | SLS cache flushed prefetch 16 b/w 0.142048 GB/s effective b/w: 0.649364GB/s bit_rate 4batch size 40 num rows 5000000 emb dim 12 avg length 40 | bit_rate 4batch size 40 num rows 5000000 emb dim 12 avg length 40 | bit_rate 4batch size 40 num rows 5000000 emb dim 12 avg length 40 64 bit indices with prefetching, lengths_sum 1600 | 64 bit indices with prefetching, lengths_sum 1600 | 64 bit indices with prefetching, lengths_sum 1600 SLS cache not flushed prefetch 4 b/w 0.175974 GB/s effective b/w: 1.12623GB/s | SLS cache not flushed prefetch 8 b/w 0.157427 GB/s effective b/w: 1.00754GB/s | SLS cache not flushed prefetch 16 b/w 0.175214 GB/s effective b/w: 1.12137GB/s SLS cache flushed prefetch 4 b/w 0.160466 GB/s effective b/w: 1.02699GB/s | SLS cache flushed prefetch 8 b/w 0.152678 GB/s effective b/w: 0.97714GB/s | SLS cache flushed prefetch 16 b/w 0.168301 GB/s effective b/w: 1.07712GB/s ``` Reviewed By: jianyuh Differential Revision: D20799658 fbshipit-source-id: cd486d1bac56662de54960237d5fd8e3e9ba6822