pytorch

cdf4d42c - [RELAND] [RELAND] .circleci: Improve docker image build workflow (#38335)

[RELAND] [RELAND] .circleci: Improve docker image build workflow (#38335) Summary: This reverts commit 6e66e8562f276e2015af8ff76437a3f0277c4bcc. Two things learned from the previous reland: * `cirlceci-agent step halt` doesn't actually halt the step in place, you must explicitly exit the step after the `step halt` is called * Even though `circleci` uses `git` to checkout repositories inside of docker images, that does not mean `git` is available after the fact. <details> <summary> Changes from previous reland </summary> ```patch commit cc99a12c9029472bd73325876bc0e9dbb1746b05 Author: Eli Uriegas <eliuriegas@fb.com> Date: Tue May 12 10:58:18 2020 -0700 .cirlceci: Install git for gc, exit step explicitly Signed-off-by: Eli Uriegas <eliuriegas@fb.com> diff --git a/.circleci/config.yml b/.circleci/config.yml index 481d7889da..856a0fb10a 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -2018,13 +2018,15 @@ jobs: export AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_DOCKER_BUILDER_V1} eval $(aws ecr get-login --no-include-email --region us-east-1) set -x + PREVIOUS_DOCKER_TAG=$(git rev-parse "$(git merge-base HEAD << pipeline.git.base_revision >>):.circleci/docker") # Check if image already exists, if it does then skip building it if docker manifest inspect "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/${IMAGE_NAME}:${DOCKER_TAG}"; then circleci-agent step halt + # circleci-agent step halt doesn't actually halt the step so we need to + # explicitly exit the step here ourselves before it causes too much trouble + exit 0 fi - PREVIOUS_DOCKER_TAG=$(git rev-parse "$(git merge-base HEAD << pipeline.git.base_revision >>):.circleci/docker") # If no image exists but the hash is the same as the previous hash then we should error out here - # no stampeding herd effect plz. if [[ ${PREVIOUS_DOCKER_TAG} = ${DOCKER_TAG} ]]; then echo "ERROR: Something has gone wrong and the previous image isn't available for the merge-base of your branch" echo " contact the PyTorch team to restore the original images" diff --git a/.circleci/ecr_gc_docker/Dockerfile b/.circleci/ecr_gc_docker/Dockerfile index d0198acb86..36347d5e6d 100644 --- a/.circleci/ecr_gc_docker/Dockerfile +++ b/.circleci/ecr_gc_docker/Dockerfile @@ -1,6 +1,6 @@ FROM ubuntu:16.04 -RUN apt-get update && apt-get install -y python-pip && rm -rf /var/lib/apt/lists/* /var/log/dpkg.log +RUN apt-get update && apt-get install -y git python-pip && rm -rf /var/lib/apt/lists/* /var/log/dpkg.log ADD requirements.txt /requirements.txt diff --git a/.circleci/verbatim-sources/docker_jobs.yml b/.circleci/verbatim-sources/docker_jobs.yml index e04d11c5cd..3918cc04ae 100644 --- a/.circleci/verbatim-sources/docker_jobs.yml +++ b/.circleci/verbatim-sources/docker_jobs.yml @@ -35,13 +35,15 @@ export AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_DOCKER_BUILDER_V1} eval $(aws ecr get-login --no-include-email --region us-east-1) set -x + PREVIOUS_DOCKER_TAG=$(git rev-parse "$(git merge-base HEAD << pipeline.git.base_revision >>):.circleci/docker") # Check if image already exists, if it does then skip building it if docker manifest inspect "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/${IMAGE_NAME}:${DOCKER_TAG}"; then circleci-agent step halt + # circleci-agent step halt doesn't actually halt the step so we need to + # explicitly exit the step here ourselves before it causes too much trouble + exit 0 fi - PREVIOUS_DOCKER_TAG=$(git rev-parse "$(git merge-base HEAD << pipeline.git.base_revision >>):.circleci/docker") # If no image exists but the hash is the same as the previous hash then we should error out here - # no stampeding herd effect plz. if [[ ${PREVIOUS_DOCKER_TAG} = ${DOCKER_TAG} ]]; then echo "ERROR: Something has gone wrong and the previous image isn't available for the merge-base of your branch" echo " contact the PyTorch team to restore the original images" ``` </details> Pull Request resolved: https://github.com/pytorch/pytorch/pull/38335 Differential Revision: D21536269 Pulled By: seemethere fbshipit-source-id: 5577f84fa49dd6e1e88fce461646fd68be3d417d