Unrolled build for rust-lang#124663

Rollup merge of rust-lang#124663 - Kobzol:docker-local-download, r=Mark-Simulacrum

Enable reusing CI Docker cache when running CI images locally

When running a CI image locally, e.g. using `DEPLOY=1 src/ci/docker/run.sh dist-x86_64-linux`, it can take a long time until the Docker image is built, which is annoying.

Since we now use proper Docker caching on CI, it should be possible to just `docker pull` the prebuilt image to reuse the cache. We didn't want to do this on CI, since our caching key isn't perfect and it's possible that we can miss some changes, but I think that for local usage it is fine (we could introduce some env. var. to force disable the image download, if needed).

r? `@Mark-Simulacrum`

src/ci/docker/run.sh

@@ -50,39 +50,35 @@ fi
 CACHE_DOMAIN="${CACHE_DOMAIN:-ci-caches.rust-lang.org}"
 
 if [ -f "$docker_dir/$image/Dockerfile" ]; then
- if isCI; then
- hash_key=/tmp/.docker-hash-key.txt
- rm -f "${hash_key}"
- echo $image >> $hash_key
-
- cat "$docker_dir/$image/Dockerfile" >> $hash_key
- # Look for all source files involves in the COPY command
- copied_files=/tmp/.docker-copied-files.txt
- rm -f "$copied_files"
- for i in $(sed -n -e '/^COPY --from=/! s/^COPY \(.*\) .*$/\1/p' \
- "$docker_dir/$image/Dockerfile"); do
- # List the file names
- find "$script_dir/$i" -type f >> $copied_files
- done
- # Sort the file names and cat the content into the hash key
- sort $copied_files | xargs cat >> $hash_key
-
- # Include the architecture in the hash key, since our Linux CI does not
- # only run in x86_64 machines.
- uname -m >> $hash_key
-
- docker --version >> $hash_key
-
- # Include cache version. Can be used to manually bust the Docker cache.
- echo "2" >> $hash_key
-
- echo "Image input"
- cat $hash_key
-
- cksum=$(sha512sum $hash_key | \
- awk '{print $1}')
- echo "Image input checksum ${cksum}"
- fi
+ hash_key=/tmp/.docker-hash-key.txt
+ rm -f "${hash_key}"
+ echo $image >> $hash_key
+
+ cat "$docker_dir/$image/Dockerfile" >> $hash_key
+ # Look for all source files involves in the COPY command
+ copied_files=/tmp/.docker-copied-files.txt
+ rm -f "$copied_files"
+ for i in $(sed -n -e '/^COPY --from=/! s/^COPY \(.*\) .*$/\1/p' \
+ "$docker_dir/$image/Dockerfile"); do
+ # List the file names
+ find "$script_dir/$i" -type f >> $copied_files
+ done
+ # Sort the file names and cat the content into the hash key
+ sort $copied_files | xargs cat >> $hash_key
+
+ # Include the architecture in the hash key, since our Linux CI does not
+ # only run in x86_64 machines.
+ uname -m >> $hash_key
+
+ # Include cache version. Can be used to manually bust the Docker cache.
+ echo "2" >> $hash_key
+
+ echo "Image input"
+ cat $hash_key
+
+ cksum=$(sha512sum $hash_key | \
+ awk '{print $1}')
+ echo "Image input checksum ${cksum}"
 
  dockerfile="$docker_dir/$image/Dockerfile"
  if [ -x /usr/bin/cygpath ]; then
@@ -105,10 +101,18 @@ if [ -f "$docker_dir/$image/Dockerfile" ]; then
  # It seems that it cannot be the same as $IMAGE_TAG, otherwise it overwrites the cache
  CACHE_IMAGE_TAG=${REGISTRY}/${REGISTRY_USERNAME}/rust-ci-cache:${cksum}
 
- # On non-CI jobs, we don't do any caching.
+ # On non-CI jobs, we try to download a pre-built image from the rust-lang-ci
+ # ghcr.io registry. If it is not possible, we fall back to building the image
+ # locally.
  if ! isCI;
  then
- retry docker build --rm -t rust-ci -f "$dockerfile" "$context"
+ if docker pull "${IMAGE_TAG}"; then
+ echo "Downloaded Docker image from CI"
+ docker tag "${IMAGE_TAG}" rust-ci
+ else
+ echo "Building local Docker image"
+ retry docker build --rm -t rust-ci -f "$dockerfile" "$context"
+ fi
  # On PR CI jobs, we don't have permissions to write to the registry cache,
  # but we can still read from it.
  elif [[ "$PR_CI_JOB" == "1" ]];