Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replication failure 504 Gateway Time-out intermittently #12372

Closed
YakirShriker opened this issue Jul 1, 2020 · 1 comment
Closed

Replication failure 504 Gateway Time-out intermittently #12372

YakirShriker opened this issue Jul 1, 2020 · 1 comment

Comments

@YakirShriker
Copy link

Harbor v2.0 over k8s installed with Helm chart(Nginx default settings), about 50% of the replication fails with this error:

image

2020-07-01T07:56:18Z [ERROR] [/replication/transfer/image/transfer.go:318]: failed to push manifest of artifact lpapps_release/lp-teamcitymetrics-app:1.0.0.10-33: http status code: 504, body: <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>
2020-07-01T07:56:18Z [ERROR] [/replication/transfer/image/transfer.go:163]: http status code: 504, body: <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>

Nginx logs:

[01/Jul/2020:07:51:59 +0000]:10.176.208.13 - "GET /service/token?scope=repository%3Alpapps_release%2Flp-teamcitymetrics-app%3Apull&service=harbor-registry HTTP/1.1" 200 737 "-" "Go-http-client/1.1" 1.547 1.548 .
[01/Jul/2020:07:51:59 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/manifests/1.0.0.10-33 HTTP/1.1" 404 0 "-" "harbor-registry-client" 0.032 0.032 .
[01/Jul/2020:07:52:00 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:6e636d7d1c6dbd5ffa84778978e1313df075b91b16c60abf1193f3d3e7d0dcb0 HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.132 0.132 .
[01/Jul/2020:07:52:00 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.129 0.128 .
[01/Jul/2020:07:52:00 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:b44464bb51b65137fb8a20e4db7f6d56f785175d3fe834126c4eadec507016c7 HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.134 0.136 .
[01/Jul/2020:07:52:00 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:47ed8ce4e6c3df46c0a6b9b1962ead20929a1bef8a6d99c890590bbfcfac7a94 HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.139 0.140 .
[01/Jul/2020:07:52:00 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:bf4ca4b9dc0325157c4d4d78ab504a8f191aa4274f7a03d27d084ef27b63ef93 HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.131 0.128 .
[01/Jul/2020:07:52:00 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:bd36c8a5ea64f9337ec0be6cfcbab6a817e6e56351274f77839fb254282239fe HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.131 0.132 .
[01/Jul/2020:07:52:00 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:c1fefbeb45ca7b123c4214ba665fa5b44050555ecb57f584597899e00c65d679 HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.125 0.128 .
[01/Jul/2020:07:52:01 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:f8c224903ee75d256060c0d88e487d740d4097185ddd03b1a38a598ef59f3b3b HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.131 0.132 .
[01/Jul/2020:07:52:01 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:b9dd315da580afa5e553d9fde7c03f5ed9ff63618a229a4913ac7324fb91a10d HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.126 0.124 .
[01/Jul/2020:07:52:01 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:729f5a5ecfe2e801e6006bd20e4c0928e5f70361af37abf039586b342985d8c9 HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.127 0.128 .
[01/Jul/2020:07:52:01 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:e882f6ff039e900c15883a99677711eb563565e00735106f2ddd9faed2288d9f HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.127 0.128 .
[01/Jul/2020:07:52:01 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:9f2e351547f9a9b15ab00fdcb2c1f50d387ca15a37b602aa4a3dab5e0db08de8 HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.127 0.124 .
[01/Jul/2020:07:52:01 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:2ceb7d6e11b952bfe64c5219038d99331c0e4743ecdbc0e3610675ef0747f543 HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.127 0.124 .
[01/Jul/2020:07:52:01 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:807bc6e44da26f19c4209229b9967362dd904e2dc1e5f1997f25b66a88aa720d HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.129 0.132 .
[01/Jul/2020:07:52:02 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:0f49d109d0cb8b6f66a9adc0eec04ac74fb1811a09b406844b235bfd84674e30 HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.124 0.124 .
[01/Jul/2020:07:52:02 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:ae19b56ac53b5d29e202ea46d7b17a8b0d7b8e6f4662dc28984ca608b287a1ce HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.133 0.132 .
[01/Jul/2020:07:52:02 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:2bab1d5390a1ebdf8fbea25c7f62ad67add78678b9339a9cc2590cd00c56191a HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.210 0.208 .
[01/Jul/2020:07:52:03 +0000]:10.176.208.13 - "GET /service/token?scope=repository%3Alpapps_release%2Flp-teamcitymetrics-app%3Apull%2Cpush&service=harbor-registry HTTP/1.1" 200 738 "-" "Go-http-client/1.1" 1.540 1.537 .
2020/07/01 07:53:03 [error] 6#0: *255047 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.176.208.13, server: , request: "PUT /v2/lpapps_release/lp-teamcitymetrics-app/manifests/1.0.0.10-33 HTTP/1.1", upstream: "http://10.176.206.81:80/v2/lpapps_release/lp-teamcitymetrics-app/manifests/1.0.0.10-33", host: "me-dcr-k8s.int.liveperson.net"
[01/Jul/2020:07:53:03 +0000]:10.176.208.13 - "PUT /v2/lpapps_release/lp-teamcitymetrics-app/manifests/1.0.0.10-33 HTTP/1.1" 504 160 "-" "harbor-registry-client" 59.997 60.003 .
[01/Jul/2020:07:53:04 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/manifests/1.0.0.3-16 HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.164 0.164 .
[01/Jul/2020:07:53:04 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/manifests/1.0.0.11-35 HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.127 0.124 .
[01/Jul/2020:07:55:14 +0000]:10.176.208.13 - "GET /service/token?scope=repository%3Alpapps_release%2Flp-teamcitymetrics-app%3Apull&service=harbor-registry HTTP/1.1" 200 737 "-" "Go-http-client/1.1" 1.560 1.560 .
[01/Jul/2020:07:55:14 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/manifests/1.0.0.10-33 HTTP/1.1" 404 0 "-" "harbor-registry-client" 0.047 0.048 .
[01/Jul/2020:07:55:14 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:6e636d7d1c6dbd5ffa84778978e1313df075b91b16c60abf1193f3d3e7d0dcb0 HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.125 0.124 .
[01/Jul/2020:07:55:14 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.208 0.208 .
[01/Jul/2020:07:55:14 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:b44464bb51b65137fb8a20e4db7f6d56f785175d3fe834126c4eadec507016c7 HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.197 0.196 .
[01/Jul/2020:07:55:15 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:47ed8ce4e6c3df46c0a6b9b1962ead20929a1bef8a6d99c890590bbfcfac7a94 HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.128 0.128 .
[01/Jul/2020:07:55:15 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:bf4ca4b9dc0325157c4d4d78ab504a8f191aa4274f7a03d27d084ef27b63ef93 HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.130 0.132 .
[01/Jul/2020:07:55:15 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:bd36c8a5ea64f9337ec0be6cfcbab6a817e6e56351274f77839fb254282239fe HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.131 0.132 .
[01/Jul/2020:07:55:15 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:c1fefbeb45ca7b123c4214ba665fa5b44050555ecb57f584597899e00c65d679 HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.131 0.132 .
[01/Jul/2020:07:55:15 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:f8c224903ee75d256060c0d88e487d740d4097185ddd03b1a38a598ef59f3b3b HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.127 0.124 .
[01/Jul/2020:07:55:15 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:b9dd315da580afa5e553d9fde7c03f5ed9ff63618a229a4913ac7324fb91a10d HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.139 0.140 .
[01/Jul/2020:07:55:15 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:729f5a5ecfe2e801e6006bd20e4c0928e5f70361af37abf039586b342985d8c9 HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.123 0.124 .
[01/Jul/2020:07:55:16 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:e882f6ff039e900c15883a99677711eb563565e00735106f2ddd9faed2288d9f HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.130 0.132 .
[01/Jul/2020:07:55:16 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:9f2e351547f9a9b15ab00fdcb2c1f50d387ca15a37b602aa4a3dab5e0db08de8 HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.130 0.128 .
[01/Jul/2020:07:55:16 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:2ceb7d6e11b952bfe64c5219038d99331c0e4743ecdbc0e3610675ef0747f543 HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.130 0.128 .
[01/Jul/2020:07:55:16 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:807bc6e44da26f19c4209229b9967362dd904e2dc1e5f1997f25b66a88aa720d HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.127 0.128 .
[01/Jul/2020:07:55:16 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:0f49d109d0cb8b6f66a9adc0eec04ac74fb1811a09b406844b235bfd84674e30 HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.125 0.124 .
[01/Jul/2020:07:55:16 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:ae19b56ac53b5d29e202ea46d7b17a8b0d7b8e6f4662dc28984ca608b287a1ce HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.128 0.128 .
[01/Jul/2020:07:55:16 +0000]:10.176.208.13 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/blobs/sha256:2bab1d5390a1ebdf8fbea25c7f62ad67add78678b9339a9cc2590cd00c56191a HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.124 0.124 .
[01/Jul/2020:07:55:18 +0000]:10.176.208.13 - "GET /service/token?scope=repository%3Alpapps_release%2Flp-teamcitymetrics-app%3Apull%2Cpush&service=harbor-registry HTTP/1.1" 200 740 "-" "Go-http-client/1.1" 1.545 1.548 .
**2020/07/01 07:56:18 [error] 6#0: *256889 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.176.208.13, server: , request: "PUT /v2/lpapps_release/lp-teamcitymetrics-app/manifests/1.0.0.10-33 HTTP/1.1", upstream: "http://10.176.206.81:80/v2/lpapps_release/lp-teamcitymetrics-app/manifests/1.0.0.10-33", host: "me-dcr-k8s.int.liveperson.net"**
[01/Jul/2020:07:56:18 +0000]:10.176.208.13 - "PUT /v2/lpapps_release/lp-teamcitymetrics-app/manifests/1.0.0.10-33 HTTP/1.1" 504 160 "-" "harbor-registry-client" 60.004 60.004 .
[01/Jul/2020:07:56:18 +0000]:10.176.208.4 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/manifests/1.0.0.3-16 HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.128 0.128 .
[01/Jul/2020:07:56:19 +0000]:10.176.208.4 - "HEAD /v2/lpapps_release/lp-teamcitymetrics-app/manifests/1.0.0.11-35 HTTP/1.1" 200 0 "-" "harbor-registry-client" 0.122 0.124 .

Seems like Nginx closing the connection.
any suggestions?
@YakirShriker
Copy link
Author

same problem as #3446
solved with increasing Nginx timeout parameters

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@YakirShriker