Pushing the image to Docker Hub takes a long time #1970
Milestone
Comments
|
Hi @bmuschko, I'm really intrigued. I assume you use Could you run with Can you also try Jib 1.4.0 to see if there's any difference? |
|
Hi, I have been struggling with this issue for the last two days. I tried the same maven build with 1.4.0 and it worked! The problem only occurs if we tag our docker images with a release. If the docker image has a tag that ends with "SNAPSHOT" it pushes just fine even with versions 1.5.1 or 1.5.0. I hope that information helps. Thank you! |
|
@kalapraveen-mi are you using And the "SNAPSHOT" tag is really weird and interesting. |
|
@kalapraveen-mi The issue you are describing is likely not related to my issue. I'd suggest opening a new GitHub issue. I have no problems with pushing images. It's about the execution time needed to perform the operation. @chanseokoh Please see the output below. Potentially a memory contention issue on the Gradle daemon process when you are trying to upload the Docker image? log (click to expand) |
|
@bmuschko the log shows that, e.g., a fairly small Blob of size ~34MB is taking a few minutes to push. I'd like to check first if pushing a BLOb manually on the command line is slow as well. To do so, you need to make three HTTP calls (curl commands) like the script below. This is basically what Jib does. Edit some variables at the top (Docker Hub account and BLOb path). #!/bin/sh
set -o errexit
set -o pipefail
# Your Docker Hub image repository
REPO=bmuschko/todo-web-service
USERNAME="your Docker Hub account"
PASSWORD="your password"
# BLOb to push. You can just use any large file. The name, path, and contents
# don't matter at all. For example, you can generate a 40MB file by
# $ dd if=/dev/urandom of=./path/to/40mb-file bs=1048576 count=40
BLOB_PATH=./path/to/40mb-file
#VERBOSE=-v
if [ ! -f "${BLOB_PATH}" ]; then
echo "Halting: ${BLOB_PATH} does not exist."
exit 1
fi
echoGreen() {
echo "$(tput setaf 2; tput bold)$1$(tput sgr0)"
}
################################################################################
# STEP 1: Auth with Docker Hub for push
################################################################################
AUTH=$( echo -n "${USERNAME}:${PASSWORD}" | base64 - )
JSON_OUTPUT=$( curl ${VERBOSE} --compressed \
-H 'Accept: */*' -H 'Accept-Encoding: gzip' \
-H "Authorization: Basic ${AUTH}" \
-H 'User-Agent: jib 1.5.1 jib-maven-plugin Google-HTTP-Java-Client/1.30.0 (gzip)' \
-- "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${REPO}:pull,push" )
# The above will return an auth token in JSON. For example:
# {
# "token": "THIS TOKEN IS WHAT WE WANT TO EXTRACT",
# "access_token": "...",
# "expires_in": 300,
# "issued_at": "2019-09-11T15:26:24.113210766Z"
# }
# (Seems like Docker Hub tokens expire in 5 minutes.)
# Extract the "token" field.
TOKEN=$( echo "${JSON_OUTPUT}" | jq -r '.token' )
echo
echoGreen ">>> Got token (don't reveal this in public): ${TOKEN}"
echo
################################################################################
# STEP 2: Ask Docker Hub where I can push a layer (HTTP POST)
################################################################################
HEADER_OUTPUT=$( curl ${VERBOSE} -I --compressed -X POST \
-H 'Accept: ' -H 'Accept-Encoding: gzip' \
-H "Authorization: Bearer ${TOKEN}" \
-H 'User-Agent: jib 1.5.1 jib-maven-plugin Google-HTTP-Java-Client/1.30.0 (gzip)' \
-- "https://registry-1.docker.io/v2/${REPO}/blobs/uploads/" )
# Server will return where to push a BLOb. For example,
# < Location: https://registry-1.docker.io/v2/<your repo>/blobs/uploads/...
PUSH_LOCATION=$( echo "${HEADER_OUTPUT}" \
| grep '^Location: ' | tr -d '\r' | sed 's/Location: //' )
echo
echoGreen ">>> Got push location: "${PUSH_LOCATION}
echo
################################################################################
# STEP 3: Push a BLOb (HTTP PATCH)
###############################################################################
echo "Now pushing a BLOb: ${BLOB_PATH}"
echo
# Now we are actually going to push a large BLOb (layer). Let's time it.
time curl ${VERBOSE} --compressed -X PATCH \
-H 'Accept: ' -H 'Accept-Encoding: gzip' \
-H "Authorization: Bearer ${TOKEN}" \
-H 'User-Agent: jib 1.5.1 jib-maven-plugin Google-HTTP-Java-Client/1.30.0 (gzip)' \
-H 'Content-Type: application/octet-stream' \
--data-binary "@${BLOB_PATH}" \
-- "${PUSH_LOCATION}"The output should be like below, showing that it took about 3 seconds for me. |
|
Oh, actually, try to run the script with different BLObs in parallel (maybe run 4~5 at once) to simulate concurrent pushing to Docker Hub. |
|
Seems like I am having issues with generating the access token. It says BTW the same upload process is much faster with the Gradle plugin version 1.4.0 though I wouldn't expect it to take a minute. |
|
I believe I can reproduce this at home where my network is very slow. With both Maven and Gradle. I've spent quite some time, monitoring network throughput on my laptop. I am reasonably sure the problem starts to happen right after the commit to upgrade the underlying Google HTTP Client library to the new v2 world. The commit is after the 1.4.0 release, so 1.4.0 should work fine as before. Multiple users already reported 1.4.0 works. |
This was referenced Sep 12, 2019
|
Forgot to mention that most of the time, I'm hitting worse than slow. Almost all the time, my builds hang indefinitely. |
|
Just joining the train of folks reporting that a downgrade to 1.4.0 fixed slow transfer speeds. Thanks, everyone! Glad we collectively cracked this one! |
|
@chanseokoh I have experienced that when i need to use a new base image for my old containers (1.5.0).
|
@chanseokoh No, I am not using any such tag and logged into Docker container on Mac. We are using Jib plugin and is run as part of "deploy" phase in the following way: Now, when I run the build of the project version contains SNAPSHOT it gets pushed effortlessly as it always does. However, if the ${project.version} contains a release it struggles and ends up with a credentials error. The same code works if I explicitly specify the version 1.4.0. We push to dockerhub by the way. Hope this info helps. Thank you very much! |
|
@kalapraveen-mi please open your issue in a new issue, as it does not appear to be directly related to this issue. |
|
The slow network operation issue should be fixed by #1980. We will release 1.6.0, probably tomorrow. |
|
@bmuschko @ndarilek @raizoor @kalapraveen-mi v1.6.0 has been released, which should fix this issue. |
|
@TadCordle Thanks. I tried it out and its close to the numbers I see with version 1.4.0. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Environment:
Description of the issue:
Push the image to Docker Hub initially takes extremely long (almost 5 mins). Subsequent push operations are much faster (seconds).
Expected behavior:
Pushing an image to Docker Hub should match the execution time when using Docker Engine.
Steps to reproduce:
./gradlew jibLog output:
The text was updated successfully, but these errors were encountered: