After upgrading to 1.31.0, library returns 403 while curl succeeds #795

chanseokoh · 2019-08-19T21:23:38Z

We upgraded from 1.27.0 to 1.31.0 with the new "v2" apache (google-http-client-apache-v2), and we are starting to see this problem.

Environment details

Java version:

openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-8u222-b10-1-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)

google-http-client version: 1.31.0, 1.30.0
google-http-client-apache-v2: 1.31.0, 1.30. 0

Steps to reproduce

Send GET to https://mcr.microsoft.com/v2/dotnet/core/aspnet/blobs/sha256:bbfbcd8743705b4d4d398c18abb8fa52b3204239d80ea089d0c3f3c53d5818dd. See the example code below.

Code example

import com.google.api.client.http.GenericUrl;
import com.google.api.client.http.apache.v2.ApacheHttpTransport;
import java.io.IOException;

public class Main {

  public static void main(String[] args) throws IOException {
    GenericUrl url =
        new GenericUrl(
            "https://mcr.microsoft.com/v2/dotnet/core/aspnet/blobs/sha256:bbfbcd8743705b4d4d398c18abb8fa52b3204239d80ea089d0c3f3c53d5818dd");
    new ApacheHttpTransport().createRequestFactory().buildGetRequest(url).execute();
  }
}

Stack trace

Both of the curl commands in the log below succeed on the command line. It is only the library that is failing. And the old 1.27.0 library used to work, returning 200.

Note the second curl command attempts a temporary redirect URL after getting the 307 response from the first GET, so eventually the second URL in the log will expire and stop working. To reproduce on your side, make sure to re-run the sample code to get a fresh temporary redirect URL.

Aug 19, 2019 5:09:04 PM com.google.api.client.http.HttpRequest execute
CONFIG: -------------- REQUEST  --------------
GET https://mcr.microsoft.com/v2/dotnet/core/aspnet/blobs/sha256:bbfbcd8743705b4d4d398c18abb8fa52b3204239d80ea089d0c3f3c53d5818dd
Accept-Encoding: gzip
User-Agent: Google-HTTP-Java-Client/1.30.0 (gzip)

Aug 19, 2019 5:09:04 PM com.google.api.client.http.HttpRequest execute
CONFIG: curl -v --compressed -H 'Accept-Encoding: gzip' -H 'User-Agent: Google-HTTP-Java-Client/1.30.0 (gzip)' -- 'https://mcr.microsoft.com/v2/dotnet/core/aspnet/blobs/sha256:bbfbcd8743705b4d4d398c18abb8fa52b3204239d80ea089d0c3f3c53d5818dd'
Aug 19, 2019 5:09:04 PM com.google.api.client.http.HttpResponse <init>
CONFIG: -------------- RESPONSE --------------
HTTP/1.1 307 Temporary Redirect
Server: nginx
Date: Mon, 19 Aug 2019 21:09:04 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 465
Connection: keep-alive
Access-Control-Expose-Headers: Docker-Content-Digest
Access-Control-Expose-Headers: WWW-Authenticate
Access-Control-Expose-Headers: Link
Access-Control-Expose-Headers: X-Ms-Correlation-Request-Id
Docker-Distribution-Api-Version: registry/2.0
Location: https://mcreus0.cdn.mscr.io/aba285c624a04409823b708c7a50e7b9-jttfjm99vo//docker/registry/v2/blobs/sha256/bb/bbfbcd8743705b4d4d398c18abb8fa52b3204239d80ea089d0c3f3c53d5818dd/data?P1=1566250144&P2=1&P3=1&P4=Bn0DGC6CZAgoV3jMryKIL4n1jTVxL7I0KUHJnSjSs7Y%3D&se=2019-08-19T21%3A29%3A04Z&sig=PtjXhfsKX9ZXps7NqNqHQpKOinCONSjOUcXJWnibGWM%3D&sp=r&sr=b&sv=2016-05-31&regid=aba285c624a04409823b708c7a50e7b9
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: 245b63cd-0445-43e2-b33d-de3467e7a820
Strict-Transport-Security: max-age=31536000; includeSubDomains

Aug 19, 2019 5:09:04 PM com.google.api.client.http.HttpRequest execute
CONFIG: -------------- REQUEST  --------------
GET https://mcreus0.cdn.mscr.io/aba285c624a04409823b708c7a50e7b9-jttfjm99vo//docker/registry/v2/blobs/sha256/bb/bbfbcd8743705b4d4d398c18abb8fa52b3204239d80ea089d0c3f3c53d5818dd/data?P1=1566250144&P2=1&P3=1&P4=Bn0DGC6CZAgoV3jMryKIL4n1jTVxL7I0KUHJnSjSs7Y%3D&se=2019-08-19T21:29:04Z&sig=PtjXhfsKX9ZXps7NqNqHQpKOinCONSjOUcXJWnibGWM%3D&sp=r&sr=b&sv=2016-05-31&regid=aba285c624a04409823b708c7a50e7b9
Accept-Encoding: gzip
User-Agent: Google-HTTP-Java-Client/1.30.0 (gzip)

Aug 19, 2019 5:09:04 PM com.google.api.client.http.HttpRequest execute
CONFIG: curl -v --compressed -H 'Accept-Encoding: gzip' -H 'User-Agent: Google-HTTP-Java-Client/1.30.0 (gzip)' -- 'https://mcreus0.cdn.mscr.io/aba285c624a04409823b708c7a50e7b9-jttfjm99vo//docker/registry/v2/blobs/sha256/bb/bbfbcd8743705b4d4d398c18abb8fa52b3204239d80ea089d0c3f3c53d5818dd/data?P1=1566250144&P2=1&P3=1&P4=Bn0DGC6CZAgoV3jMryKIL4n1jTVxL7I0KUHJnSjSs7Y%3D&se=2019-08-19T21:29:04Z&sig=PtjXhfsKX9ZXps7NqNqHQpKOinCONSjOUcXJWnibGWM%3D&sp=r&sr=b&sv=2016-05-31&regid=aba285c624a04409823b708c7a50e7b9'
Aug 19, 2019 5:09:04 PM com.google.api.client.http.HttpResponse <init>
CONFIG: -------------- RESPONSE --------------
HTTP/1.1 403 Forbidden
Content-Length: 45
Content-Type: application/xml
Server: Microsoft-HTTPAPI/2.0
X-Cache: TCP_MISS
x-ms-request-id: b6ab5dbd-801e-002c-19d2-562310000000
X-Azure-Ref-OriginShield: Ref A: 30815D44600D42D8BE35AE90D18846CE Ref B: BL2EDGE1016 Ref C: 2019-08-19T21:09:04Z
X-MSEdge-Ref: Ref A: E00E9277380B413D991576016B513F0D Ref B: NYCEDGE0715 Ref C: 2019-08-19T21:09:04Z
Date: Mon, 19 Aug 2019 21:09:04 GMT

Aug 19, 2019 5:09:04 PM com.google.api.client.util.LoggingByteArrayOutputStream close
CONFIG: Total: 45 bytes
Aug 19, 2019 5:09:04 PM com.google.api.client.util.LoggingByteArrayOutputStream close
CONFIG: ERROR 403: Time-Limited URL validation failed
Exception in thread "main" com.google.api.client.http.HttpResponseException: 403 Forbidden
ERROR 403: Time-Limited URL validation failed
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1094)
	at Main.main(Main.java:11)

The text was updated successfully, but these errors were encountered:

chanseokoh · 2019-08-19T21:39:35Z

1.27.0 succeeds with 200 from the same sample code. (Note the v1 com.google.api.client.http.apache.ApacheHttpTransport instead of the v2 com.google.api.client.http.apache.v2.ApacheHttpTransport).

import com.google.api.client.http.GenericUrl;
import com.google.api.client.http.apache.ApacheHttpTransport;
import java.io.IOException;

public class Main {

  public static void main(String[] args) throws IOException {
    GenericUrl url =
        new GenericUrl(
            "https://mcr.microsoft.com/v2/dotnet/core/aspnet/blobs/sha256:bbfbcd8743705b4d4d398c18abb8fa52b3204239d80ea089d0c3f3c53d5818dd");
    new ApacheHttpTransport().createRequestFactory().buildGetRequest(url).execute();
  }
}

Log:

Aug 19, 2019 5:36:32 PM com.google.api.client.http.HttpRequest execute
CONFIG: -------------- REQUEST  --------------
GET https://mcr.microsoft.com/v2/dotnet/core/aspnet/blobs/sha256:bbfbcd8743705b4d4d398c18abb8fa52b3204239d80ea089d0c3f3c53d5818dd
Accept-Encoding: gzip
User-Agent: Google-HTTP-Java-Client/1.27.0 (gzip)

Aug 19, 2019 5:36:32 PM com.google.api.client.http.HttpRequest execute
CONFIG: curl -v --compressed -H 'Accept-Encoding: gzip' -H 'User-Agent: Google-HTTP-Java-Client/1.27.0 (gzip)' -- 'https://mcr.microsoft.com/v2/dotnet/core/aspnet/blobs/sha256:bbfbcd8743705b4d4d398c18abb8fa52b3204239d80ea089d0c3f3c53d5818dd'
Aug 19, 2019 5:36:33 PM com.google.api.client.http.HttpResponse <init>
CONFIG: -------------- RESPONSE --------------
HTTP/1.1 307 Temporary Redirect
Server: nginx
Date: Mon, 19 Aug 2019 21:36:33 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 473
Connection: keep-alive
Access-Control-Expose-Headers: Docker-Content-Digest
Access-Control-Expose-Headers: WWW-Authenticate
Access-Control-Expose-Headers: Link
Access-Control-Expose-Headers: X-Ms-Correlation-Request-Id
Docker-Distribution-Api-Version: registry/2.0
Location: https://mcrweu0.cdn.mscr.io/42012bb2682a4d76ba7fa17a9d9a9162-qb2vm9uiex//docker/registry/v2/blobs/sha256/bb/bbfbcd8743705b4d4d398c18abb8fa52b3204239d80ea089d0c3f3c53d5818dd/data?P1=1566251793&P2=1&P3=1&P4=%2FpHFwBezXviH55LkJ5WuARZrbM%2Bj3raOr8CTeyo1JgI%3D&se=2019-08-19T21%3A56%3A33Z&sig=EIRVQYRXX4oxVU8xLVRb4A%2BXGkOKA2SlyVtf6e%2Fd9q4%3D&sp=r&sr=b&sv=2016-05-31&regid=42012bb2682a4d76ba7fa17a9d9a9162
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: 2ea40b9a-bf0e-463c-aa66-d05f7cfe9be3
Strict-Transport-Security: max-age=31536000; includeSubDomains

Aug 19, 2019 5:36:33 PM com.google.api.client.http.HttpRequest execute
CONFIG: -------------- REQUEST  --------------
GET https://mcrweu0.cdn.mscr.io/42012bb2682a4d76ba7fa17a9d9a9162-qb2vm9uiex//docker/registry/v2/blobs/sha256/bb/bbfbcd8743705b4d4d398c18abb8fa52b3204239d80ea089d0c3f3c53d5818dd/data?P1=1566251793&P2=1&P3=1&P4=/pHFwBezXviH55LkJ5WuARZrbM%2Bj3raOr8CTeyo1JgI%3D&se=2019-08-19T21:56:33Z&sig=EIRVQYRXX4oxVU8xLVRb4A%2BXGkOKA2SlyVtf6e/d9q4%3D&sp=r&sr=b&sv=2016-05-31&regid=42012bb2682a4d76ba7fa17a9d9a9162
Accept-Encoding: gzip
User-Agent: Google-HTTP-Java-Client/1.27.0 (gzip)

Aug 19, 2019 5:36:33 PM com.google.api.client.http.HttpRequest execute
CONFIG: curl -v --compressed -H 'Accept-Encoding: gzip' -H 'User-Agent: Google-HTTP-Java-Client/1.27.0 (gzip)' -- 'https://mcrweu0.cdn.mscr.io/42012bb2682a4d76ba7fa17a9d9a9162-qb2vm9uiex//docker/registry/v2/blobs/sha256/bb/bbfbcd8743705b4d4d398c18abb8fa52b3204239d80ea089d0c3f3c53d5818dd/data?P1=1566251793&P2=1&P3=1&P4=/pHFwBezXviH55LkJ5WuARZrbM%2Bj3raOr8CTeyo1JgI%3D&se=2019-08-19T21:56:33Z&sig=EIRVQYRXX4oxVU8xLVRb4A%2BXGkOKA2SlyVtf6e/d9q4%3D&sp=r&sr=b&sv=2016-05-31&regid=42012bb2682a4d76ba7fa17a9d9a9162'
Aug 19, 2019 5:36:33 PM com.google.api.client.http.HttpResponse <init>
CONFIG: -------------- RESPONSE --------------
HTTP/1.1 200 OK
Content-Length: 4061
Content-Type: application/octet-stream
Last-Modified: Thu, 15 Aug 2019 09:03:49 GMT
Accept-Ranges: bytes
ETag: "0x8D7215F7CE4676B"
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
X-Cache: TCP_MISS
x-ms-request-id: 469b87d9-d01e-0067-5ad6-561ae0000000
x-ms-version: 2016-05-31
x-ms-lease-status: unlocked
x-ms-lease-state: available
x-ms-blob-type: AppendBlob
x-ms-blob-committed-block-count: 1
x-ms-copy-id: 3ad5aa2c-f526-49cb-be97-5c7c4b2a75a0
x-ms-copy-source: https://mcrweu0.blob.core.windows.net/42012bb2682a4d76ba7fa17a9d9a9162-qb2vm9uiex//docker/registry/v2/blobs/sha256/bb/bbfbcd8743705b4d4d398c18abb8fa52b3204239d80ea089d0c3f3c53d5818dd/data/pending?sv=2017-04-17&sr=b&sig=KWKCehS%2FKUkLuKQ%2Bv9TVZFzU23TppiRYXhEEQhnit30%3D&se=2019-08-22T09%3A03%3A49Z&sp=r&api-version=2017-04-17
x-ms-copy-status: success
x-ms-copy-progress: 4061/4061
x-ms-copy-completion-time: Thu, 15 Aug 2019 09:03:49 GMT
x-ms-server-encrypted: true
X-Azure-Ref-OriginShield: Ref A: 44CCCA4FFC014BAE8EBB9B5A2777A51E Ref B: BL2EDGE0820 Ref C: 2019-08-19T21:36:33Z
X-MSEdge-Ref: Ref A: EC3140DCC5DD4F798506F04BA9C71038 Ref B: NYCEDGE0515 Ref C: 2019-08-19T21:36:33Z
Date: Mon, 19 Aug 2019 21:36:33 GMT

chanseokoh · 2019-08-19T22:04:54Z

I suspect that 1.31.0 fails to capture and send the correct temporary redirect URL from the 307 response. The server is probably working correctly by returning the "time-limited URL validation failure" for non-existing or incorrect URLs.

chingor13 · 2019-08-20T22:04:06Z

Using the legacy apache client also fails at 1.31.0 so I looked into other things that changed.

Interestingly, downgrading org.apache.httpcomponents:httpclient back to 4.5.5 and org.apache.httpcomponents:4.4.9 seems to fix the issue. I will try and narrow down what happened.

chingor13 · 2019-08-20T23:19:35Z

I narrowed it down to org.apache.httpcomponents:httpclient 4.5.6->4.5.7

chingor13 · 2019-08-21T00:01:21Z

Looks like it broke here: apache/httpcomponents-client@8c04c6a

There are additional reports on that commit that it's breaking other URL signing.

chanseokoh · 2019-08-21T15:34:50Z

Thanks for looking into it. Then what should the resolution be? According to the thread in that commit, seems like it's possible to disable the URI normalization (at least it is reported possible in 4.5.8.) Should google-http-java-client do it? Or, can we handle the breaking change from new URI normalization probably at the google-http-java-client level and make things work? Or, is it that microsoft.com should ideally be fixed to handle normalized URIs properly?

chingor13 · 2019-08-21T15:52:06Z

It looks like we need set the config here and/or provide a way to configure the request options builder at client configuration time.

chingor13 · 2019-08-21T15:53:09Z

As a workaround, until we get this patched and released, you can force downgrade the org.apache.httpcomponents:httpclient package to 4.5.6

chanseokoh · 2019-08-21T16:14:46Z

Do you think it is safe and compatible to force downgrading only the transitive dependency org.apache.httpcomponents:httpclient to 4.5.6 alone while keeping 1.31.0?

chingor13 · 2019-08-21T16:15:58Z

Yes, we weren't relying on anything added in higher versions, just trying to keep up to date dependencies.

elharo · 2019-08-30T11:39:24Z

After reading through that thread, I think the answer is that "microsoft.com should ideally be fixed to handle normalized URIs properly". I'm not 100% sure of that, but it seems the most likely answer.

What's the impact of not fixing this on our side?
Has anyone reported this issue to Microsoft?

chingor13 · 2019-08-30T17:14:20Z

It seems according to the spec that the normalization is for comparing URIs. I don't see a reason for us to need to modify the user's request URI, so we should try our best not to.
I spent about 5 minutes looking for a place to file an issue for Azure Container Registry, but couldn't find a form.

chanseokoh mentioned this issue Aug 19, 2019

After upgrading to 1.5.0 I get 403 on from image retrieval GoogleContainerTools/jib#1914

Closed

chanseokoh mentioned this issue Aug 19, 2019

1.31.0 reports wrong version in User-Agent #796

Closed

yoshi-automation added the triage me I really want to be triaged. label Aug 20, 2019

chanseokoh mentioned this issue Aug 20, 2019

Revert "Upgrade google-http-client to eliminate deprecated usage GoogleContainerTools/jib#1915

Closed

chingor13 added priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Aug 20, 2019

chingor13 self-assigned this Aug 20, 2019

yoshi-automation removed the triage me I really want to be triaged. label Aug 20, 2019

chanseokoh mentioned this issue Aug 21, 2019

Force downgrading to org.apache.httpcomponents:httpclient:4.5.6 prevent URI normalization GoogleContainerTools/jib#1924

Merged

todo bot mentioned this issue Aug 22, 2019

[pull] master from GoogleContainerTools:master daddyfatstacksBIG/jib#74

Merged

chanseokoh mentioned this issue Aug 22, 2019

Regression: connection timeout ignored in 1.31.0 while works in 1.27.0 #799

Closed

yoshi-automation added 🚨 This issue needs some love. and removed 🚨 This issue needs some love. labels Aug 26, 2019

todo bot mentioned this issue Aug 27, 2019

[pull] master from GoogleContainerTools:master daddyfatstacksBIG/jib#75

Merged

chingor13 mentioned this issue Aug 28, 2019

fix: disable uri normalization in ApacheHttpRequest #804

Merged

chingor13 closed this as completed in #804 Aug 30, 2019

chanseokoh mentioned this issue Oct 30, 2019

Option to use "Location:" header value verbatim for redirection #864

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

After upgrading to 1.31.0, library returns 403 while curl succeeds #795

After upgrading to 1.31.0, library returns 403 while curl succeeds #795

chanseokoh commented Aug 19, 2019 •

edited

Loading

chanseokoh commented Aug 19, 2019

chanseokoh commented Aug 19, 2019 •

edited

Loading

chingor13 commented Aug 20, 2019

chingor13 commented Aug 20, 2019

chingor13 commented Aug 21, 2019

chanseokoh commented Aug 21, 2019 •

edited

Loading

chingor13 commented Aug 21, 2019

chingor13 commented Aug 21, 2019

chanseokoh commented Aug 21, 2019

chingor13 commented Aug 21, 2019 •

edited

Loading

elharo commented Aug 30, 2019

chingor13 commented Aug 30, 2019 •

edited

Loading

After upgrading to 1.31.0, library returns 403 while curl succeeds #795

After upgrading to 1.31.0, library returns 403 while curl succeeds #795

Comments

chanseokoh commented Aug 19, 2019 • edited Loading

Environment details

Steps to reproduce

Code example

Stack trace

chanseokoh commented Aug 19, 2019

chanseokoh commented Aug 19, 2019 • edited Loading

chingor13 commented Aug 20, 2019

chingor13 commented Aug 20, 2019

chingor13 commented Aug 21, 2019

chanseokoh commented Aug 21, 2019 • edited Loading

chingor13 commented Aug 21, 2019

chingor13 commented Aug 21, 2019

chanseokoh commented Aug 21, 2019

chingor13 commented Aug 21, 2019 • edited Loading

elharo commented Aug 30, 2019

chingor13 commented Aug 30, 2019 • edited Loading

chanseokoh commented Aug 19, 2019 •

edited

Loading

chanseokoh commented Aug 19, 2019 •

edited

Loading

chanseokoh commented Aug 21, 2019 •

edited

Loading

chingor13 commented Aug 21, 2019 •

edited

Loading

chingor13 commented Aug 30, 2019 •

edited

Loading