pytorch

6da4a255 - Use private squid proxy (#62244)

Use private squid proxy (#62244) Summary: This PR adds a **private** squid proxy (note that the internal ELB is only accessible from the private VPC subnets of GitHub Runners) that's deployed dedicated for PyTorch CI for GitHub runners. ``` dig $SQUID_PROXY 10.0.x.x 10.0.x.x ``` http_proxy and https_proxy are compatible with the following http clients: - curl - wget - python Existing cache policy: refresh_pattern -i .(7z|deb|rpm|exe|zip|tar|tgz|gz|ram|rar|bin|tiff|bz2|run|csv|sh)$ 1440 80% 2880 It uses the standard squid refresh_pattern for cache requests. In our setup, we tried to cache at least (1440 minutes - 1 day) and at max (2880 minutes - 2 days), with last-modified factor 80% (squid doc). Please refer to pytorch/test-infra for details. Right now, it only applies to the build and test step, to limit the scope and make sure build and test are more reliable with egress cache. Pull Request resolved: https://github.com/pytorch/pytorch/pull/62244 Test Plan: ``` # first time, cache miss (4min20s) http_proxy=$SQUID_PROXY https_proxy=$SQUID_PROXY curl -v -L http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz --output /tmp/tmp_mnist.zip 100 9680k 100 9680k 0 0 37836 0 0:04:21 0:04:21 --:--:-- 29908 # second time, cache hit (0s) http_proxy=$SQUID_PROXY https_proxy=$SQUID_PROXY curl -v -L http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz --output /tmp/tmp_mnist.zip 100 9680k 100 9680k 0 0 103M 0 --:--:-- --:--:-- --:--:-- 103M ``` Load Test Plan: ``` # ab load test with `-n 100` requests ab -X $SQUID_PROXY -n 100 http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz Concurrency Level: 1 Time taken for tests: 9.044 seconds Complete requests: 100 Failed requests: 0 Total transferred: 991326300 bytes HTML transferred: 991242200 bytes Requests per second: 11.06 [#/sec] (mean) Time per request: 90.442 [ms] (mean) Time per request: 90.442 [ms] (mean, across all concurrent requests) Transfer rate: 107040.50 [Kbytes/sec] received ``` Reviewed By: malfet Differential Revision: D29928698 Pulled By: zhouzhuojie fbshipit-source-id: 4ee78be0abe35411666c6121991b0addded57106