[RELAND] .circleci: Improve docker image build workflow (#38484)
Summary:
closes https://github.com/pytorch/pytorch/issues/37855
Relies on https://github.com/pytorch/pytorch/pull/38483
Previous attempts to get this right:
* https://github.com/pytorch/pytorch/pull/38335
* https://github.com/pytorch/pytorch/pull/38279
* https://github.com/pytorch/pytorch/pull/37976
This reverts commit 80639604a82422e314890f154242202a43d264f9.
Improves the docker image build workflow from many steps to basically
transparent from a user's perspective.
To update docker images now all one has to do is edit the
.circleci/docker folder and it will update automatically and also
dynamically add the tags to the list of tags to keep from the garbage
collector.
Adding a new image will currently stay the same but we can explore doing
that dynamically as well.
How the build workflow works:
- Docker tags are determined by the hash defined from git for the
.circleci/docker sub-directory (extracted using git rev-parse)
- Images are only built if the computed hash is not found in ecr and
the hash is different than the previously computed hash. The
previously computed hash is found using the same process as before
but subbing out HEAD for the merge base between HEAD and the base
git revision
- That tag is then passed through the jobs using a shared workspace
which is added to downstream jobs using the circleci ${BASH_ENV}
How the new garbage collection works:
- Tags to keep are generated by stepping through all of the commits in
in the .circleci/docker subdirectory
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/38484
Differential Revision: D21585458
Pulled By: seemethere
fbshipit-source-id: 37792a1e0f5e5531438c4ae61507639c133aa76d
