Use private squid proxy (#62244)
Summary:
This PR adds a **private** squid proxy (note that the internal ELB is only accessible from the private VPC subnets of GitHub Runners) that's deployed dedicated for PyTorch CI for GitHub runners.
```
dig $SQUID_PROXY
10.0.x.x
10.0.x.x
```
http_proxy and https_proxy are compatible with the following http clients:
- curl
- wget
- python
Existing cache policy:
refresh_pattern -i .(7z|deb|rpm|exe|zip|tar|tgz|gz|ram|rar|bin|tiff|bz2|run|csv|sh)$ 1440 80% 2880
It uses the standard squid refresh_pattern for cache requests. In our setup, we tried
to cache at least (1440 minutes - 1 day) and at max (2880 minutes - 2 days), with
last-modified factor 80% (squid doc). Please refer to pytorch/test-infra for details.
Right now, it only applies to the build and test step, to limit the scope and make sure build and test are more reliable with egress cache.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/62244
Test Plan:
```
# first time, cache miss (4min20s)
http_proxy=$SQUID_PROXY https_proxy=$SQUID_PROXY curl -v -L http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz --output /tmp/tmp_mnist.zip
100 9680k 100 9680k 0 0 37836 0 0:04:21 0:04:21 --:--:-- 29908
# second time, cache hit (0s)
http_proxy=$SQUID_PROXY https_proxy=$SQUID_PROXY curl -v -L http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz --output /tmp/tmp_mnist.zip
100 9680k 100 9680k 0 0 103M 0 --:--:-- --:--:-- --:--:-- 103M
```
Load Test Plan:
```
# ab load test with `-n 100` requests
ab -X $SQUID_PROXY -n 100 http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
Concurrency Level: 1
Time taken for tests: 9.044 seconds
Complete requests: 100
Failed requests: 0
Total transferred: 991326300 bytes
HTML transferred: 991242200 bytes
Requests per second: 11.06 [#/sec] (mean)
Time per request: 90.442 [ms] (mean)
Time per request: 90.442 [ms] (mean, across all concurrent requests)
Transfer rate: 107040.50 [Kbytes/sec] received
```
Reviewed By: malfet
Differential Revision: D29928698
Pulled By: zhouzhuojie
fbshipit-source-id: 4ee78be0abe35411666c6121991b0addded57106
