Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

replication for big images failed with Gateway Time-out intermittently #3446

Closed
ligc opened this issue Oct 24, 2017 · 4 comments
Closed

replication for big images failed with Gateway Time-out intermittently #3446

ligc opened this issue Oct 24, 2017 · 4 comments
Assignees
@ywk253100
Labels
kind/bug kind/customer-found

Comments

@ligc
Copy link

ligc commented Oct 24, 2017

I have some docker images with a super big file of 1GB in the images, Harbor replication for these images failed intermittently with error "Gateway Time-out", if I disable and re-enable the replication rule to trigger a re-replication, it will succeed.

The Harbor is configured with 2 replication rules to replicate the images to 2 target Harbor registries.

Harbor 1.2.0
Docker 1.12.6
docker-compose 1.9.0

2017-10-23T19:27:48Z [INFO] initializing: repository: library/harbor-test-1000, tags: [latest], source URL: http://registry:5000, destination URL: http://x3.clusters.com, insecure: false, destination user: admin
2017-10-23T19:27:48Z [INFO] initialization completed: project: library, repository: library/harbor-test-1000, tags: [latest], source URL: http://registry:5000, destination URL: http://x3.clusters.com, insecure: false, destination user: admin
2017-10-23T19:27:49Z [WARNING] the status code is 409 when creating project library on http://x3.clusters.com with user admin, try to do next step
2017-10-23T19:27:49Z [INFO] manifest of library/harbor-test-1000:latest pulled successfully from http://registry:5000: sha256:87b7558465d873599d2ab1077fb7564bc70a2ccc08604ce624070f3759c76470
2017-10-23T19:27:49Z [INFO] all blobs of library/harbor-test-1000:latest from http://registry:5000: [sha256:3f6f06182add9ab3642b5912c8d94d53cec765491f1652331321aee97ce9ef52 sha256:280aca6ddce2bc2b0904fb525b14e119111c11ca06c3c4d9e4258723da52aecd sha256:fbc94b68ab231ff9418684c3e5bc09092be65c2f2b20f36883183824c9942495]
2017-10-23T19:27:49Z [INFO] blobs of library/harbor-test-1000:latest need to be transferred to http://x3.clusters.com: [sha256:3f6f06182add9ab3642b5912c8d94d53cec765491f1652331321aee97ce9ef52 sha256:280aca6ddce2bc2b0904fb525b14e119111c11ca06c3c4d9e4258723da52aecd sha256:fbc94b68ab231ff9418684c3e5bc09092be65c2f2b20f36883183824c9942495]
2017-10-23T19:27:49Z [INFO] transferring blob sha256:3f6f06182add9ab3642b5912c8d94d53cec765491f1652331321aee97ce9ef52 of library/harbor-test-1000:latest to http://x3.clusters.com ...
2017-10-23T19:27:49Z [INFO] blob sha256:3f6f06182add9ab3642b5912c8d94d53cec765491f1652331321aee97ce9ef52 of library/harbor-test-1000:latest transferred to http://x3.clusters.com completed
2017-10-23T19:27:49Z [INFO] transferring blob sha256:280aca6ddce2bc2b0904fb525b14e119111c11ca06c3c4d9e4258723da52aecd of library/harbor-test-1000:latest to http://x3.clusters.com ...
2017-10-23T19:27:50Z [INFO] blob sha256:280aca6ddce2bc2b0904fb525b14e119111c11ca06c3c4d9e4258723da52aecd of library/harbor-test-1000:latest transferred to http://x3.clusters.com completed
2017-10-23T19:27:50Z [INFO] transferring blob sha256:fbc94b68ab231ff9418684c3e5bc09092be65c2f2b20f36883183824c9942495 of library/harbor-test-1000:latest to http://x3.clusters.com ...
2017-10-23T19:29:21Z [ERROR] [transfer.go:431]: an error occurred while pushing blob sha256:fbc94b68ab231ff9418684c3e5bc09092be65c2f2b20f36883183824c9942495 of library/harbor-test-1000:latest to http://x3.clusters.com : 504 <html>

<head><title>504 Gateway Time-out</title></head>

<body bgcolor="white">

<center><h1>504 Gateway Time-out</h1></center>

<hr><center>nginx/1.11.13</center>

</body>

</html>
@ywk253100
Copy link
Contributor

Try to add timeout settings in the configuration file of Nginx.

@ligc
Copy link
Author

ligc commented Oct 31, 2017

Thank you, @ywk253100, you suggestion works perfectly.

I added the proxy_send_timeout and proxy_read_timeout in the "location /v2/" section of common/templates/nginx/nginx.http.conf; ./prepare, restart Harbor. Then the 1000MB images could be synced correctly to the remote Harbor repositories.

    location /v2/ {
      proxy_pass http://ui/registryproxy/v2/;
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      
      # When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_buffering off;
      proxy_request_buffering off;

      proxy_send_timeout 900;
      proxy_read_timeout 900;
    }


2017-10-30T18:20:48Z [INFO] initializing: repository: library/harbor-test-1000m, tags: [latest], source URL: http://registry:5000, destination URL: http://x2.clusters.com, insecure: false, destination user: admin
2017-10-30T18:20:48Z [INFO] initialization completed: project: library, repository: library/harbor-test-1000m, tags: [latest], source URL: http://registry:5000, destination URL: http://x2.clusters.com, insecure: false, destination user: admin
2017-10-30T18:20:48Z [WARNING] the status code is 409 when creating project library on http://x2.clusters.com with user admin, try to do next step
2017-10-30T18:20:48Z [INFO] manifest of library/harbor-test-1000m:latest pulled successfully from http://registry:5000: sha256:8eec0c51f62afcc3e979a059ce303be3afe0134c010d0c99c5b7063c61455d2b
2017-10-30T18:20:48Z [INFO] all blobs of library/harbor-test-1000m:latest from http://registry:5000: [sha256:38b53c28cb5d501c1dcd09e9cbe163c52a7eba488c9c7c64bb7b10a88e4755d5 sha256:30064267e5b89281ce906c124096c47516d73ac7219e30d678b934eedc93836c sha256:578dd27b81aad17150d6f375c1f04f55ea789288daf39909e5a6e61c87ea3b34]
2017-10-30T18:20:49Z [INFO] blobs of library/harbor-test-1000m:latest need to be transferred to http://x2.clusters.com: [sha256:38b53c28cb5d501c1dcd09e9cbe163c52a7eba488c9c7c64bb7b10a88e4755d5 sha256:30064267e5b89281ce906c124096c47516d73ac7219e30d678b934eedc93836c sha256:578dd27b81aad17150d6f375c1f04f55ea789288daf39909e5a6e61c87ea3b34]
2017-10-30T18:20:49Z [INFO] transferring blob sha256:38b53c28cb5d501c1dcd09e9cbe163c52a7eba488c9c7c64bb7b10a88e4755d5 of library/harbor-test-1000m:latest to http://x2.clusters.com ...
2017-10-30T18:20:49Z [INFO] blob sha256:38b53c28cb5d501c1dcd09e9cbe163c52a7eba488c9c7c64bb7b10a88e4755d5 of library/harbor-test-1000m:latest transferred to http://x2.clusters.com completed
2017-10-30T18:20:49Z [INFO] transferring blob sha256:30064267e5b89281ce906c124096c47516d73ac7219e30d678b934eedc93836c of library/harbor-test-1000m:latest to http://x2.clusters.com ...
2017-10-30T18:20:50Z [INFO] blob sha256:30064267e5b89281ce906c124096c47516d73ac7219e30d678b934eedc93836c of library/harbor-test-1000m:latest transferred to http://x2.clusters.com completed
2017-10-30T18:20:50Z [INFO] transferring blob sha256:578dd27b81aad17150d6f375c1f04f55ea789288daf39909e5a6e61c87ea3b34 of library/harbor-test-1000m:latest to http://x2.clusters.com ...
2017-10-30T18:22:25Z [INFO] blob sha256:578dd27b81aad17150d6f375c1f04f55ea789288daf39909e5a6e61c87ea3b34 of library/harbor-test-1000m:latest transferred to http://x2.clusters.com completed
2017-10-30T18:22:25Z [INFO] manifest of library/harbor-test-1000m:latest exists on source registry http://registry:5000, continue manifest pushing
2017-10-30T18:22:25Z [INFO] manifest of library/harbor-test-1000m:latest has been pushed to http://x2.clusters.com
2017-10-30T18:22:25Z [INFO] no tag needs to be replicated, next state is "finished"

I am fine with closing this issue, not sure if Harbor project wants to document this as a tip of replicating big images or the network between the replicated Harbor repositories is not good enough.

@ywk253100
Copy link
Contributor

@ligc Thank you for your suggestion. I have added it to the wiki page here. And I'm closing this issue.

@moooofly
Copy link

FYI, the above resolution not works for me.

But it dose work if I add the timeout options into server block.

@lf1029698952 lf1029698952 mentioned this issue Apr 19, 2019
Closed
@rafis rafis mentioned this issue Feb 28, 2020
Closed
@YakirShriker YakirShriker mentioned this issue Jul 1, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ywk253100 ywk253100

Labels
kind/bug kind/customer-found
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@wy65701436 @ligc @ywk253100 @moooofly