After upgrading to 1.5.0 I get 403 on from image retrieval

[tandeday]

Environment:

• Jib version: 1.5.0
• Build tool: mvn 3.6.1
• OS: Windows 10

Description of the issue:
After upgrading from 1.4.0 to 1.5.0 I get the following error when trying to create a docker build,

[INFO] Containerizing application to Docker daemon as microsvc-operational-information...
[INFO] Executing tasks:
[INFO] [=========                     ] 30,0% complete
[INFO] > pulling base image manifest
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  4.477 s
[INFO] Finished at: 2019-08-19T10:48:23+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:1.5.0:dockerBuild (default-cli) on project operational-information-ms: com.google.cloud.tools.jib.api.RegistryUnauthorizedException: Unauthorized for mcr.microsoft.com/dotnet/core/aspnet: 403 Forbidden
[ERROR] ERROR 403: Time-Limited URL validation failed
[ERROR] -> [Help 1]
[ERROR]

Downgrading to 1.4.0 again runs as before.

Expected behavior:

Build executes normally.

Steps to reproduce:

1. Run mvn jib:dockerBuild with image mcr.microsoft.com/dotnet/core/aspnet:2.1-stretch-slim
2. Build fails.

jib-maven-plugin Configuration:

            <plugin>
                <!-- https:/GoogleContainerTools/jib/tree/master/jib-maven-plugin -->
                <groupId>com.google.cloud.tools</groupId>
                <artifactId>jib-maven-plugin</artifactId>
                <version>1.5.0</version>
                <configuration>
                    <container>
                        <!-- https:/GoogleContainerTools/jib/tree/master/jib-maven-plugin#container-object -->
                        <entrypoint>
                            <command>dotnet</command>
                            <arg>Operational_Information.dll</arg>
                        </entrypoint>
                        <ports>
                            <port>80</port>
                            <port>443</port>
                        </ports>
			<user>1001</user>
                        <workingDirectory>/netcoreapp2.1/publish</workingDirectory>
			<creationTime>USE_CURRENT_TIMESTAMP</creationTime>
                    </container>
                    <from>
                        <!-- https://hub.docker.com/_/microsoft-dotnet-core -->
                        <image>mcr.microsoft.com/dotnet/core/aspnet:2.1-stretch-slim</image>
                    </from>
                    <to>
                        <image>${docker.image.local.name}</image>
                    </to>
                    <extraDirectories>
                        <paths>
                            <path>Operational_Information/bin/Release/</path>
                        </paths>
                    </extraDirectories>
                    <dockerClient>
                        <environment>
                            <key3>value3</key3>
                            <key4>value4</key4>
                        </environment>
                    </dockerClient>
                    <!-- for now, later require cert enhanced JVM -->
                    <allowInsecureRegistries>true</allowInsecureRegistries>
                </configuration>
            </plugin>

Log output:

Additional Information:

[chanseokoh]

This is easily reproducible. Affects every build goal. Looking into it now.

[chanseokoh]

A log shows that the following request returns 403. But what is strange is that running the curl command as shown in the log on the command line succeeds and returns 200. (It will eventually fail when the time-limited temporary redirect URL expires.)

Aug 19, 2019 4:01:30 PM com.google.api.client.http.HttpRequest execute
CONFIG: -------------- REQUEST  --------------
GET https://mcreus0.cdn.mscr.io/aba285c624a04409823b708c7a50e7b9-jttfjm99vo//docker/registry/v2/blobs/sha256/bb/bbfbcd8743705b4d4d398c18abb8fa52b3204239d80ea089d0c3f3c53d5818dd/data?P1=1566246090&P2=1&P3=1&P4=0Xx7W46aHEWxKc2asAWPwGnIZvimsauQgUQpdNUe%2B7I%3D&se=2019-08-19T20:21:30Z&sig=hHsVVuS3nafQi/hfaXerl45JS4bCEWVDA7bKWIDWsGI%3D&sp=r&sr=b&sv=2016-05-31&regid=aba285c624a04409823b708c7a50e7b9
Accept:
Accept-Encoding: gzip
User-Agent: jib 1.5.0 jib-maven-plugin Google-HTTP-Java-Client/1.30.0 (gzip)

Aug 19, 2019 4:01:30 PM com.google.api.client.http.HttpRequest execute
CONFIG: curl -v --compressed -H 'Accept: ' -H 'Accept-Encoding: gzip' -H 'User-Agent: jib 1.5.0 jib-maven-plugin Google-HTTP-Java-Client/1.30.0 (gzip)' -- 'https://mcreus0.cdn.mscr.io/aba285c624a04409823b708c7a50e7b9-jttfjm99vo//docker/registry/v2/blobs/sha256/bb/bbfbcd8743705b4d4d398c18abb8fa52b3204239d80ea089d0c3f3c53d5818dd/data?P1=1566246090&P2=1&P3=1&P4=0Xx7W46aHEWxKc2asAWPwGnIZvimsauQgUQpdNUe%2B7I%3D&se=2019-08-19T20:21:30Z&sig=hHsVVuS3nafQi/hfaXerl45JS4bCEWVDA7bKWIDWsGI%3D&sp=r&sr=b&sv=2016-05-31&regid=aba285c624a04409823b708c7a50e7b9'
Aug 19, 2019 4:01:30 PM com.google.api.client.http.HttpResponse <init>
CONFIG: -------------- RESPONSE --------------
HTTP/1.1 403 Forbidden
Content-Length: 45
Content-Type: application/xml
Server: Microsoft-HTTPAPI/2.0
X-Cache: TCP_MISS
x-ms-request-id: cf99d1f1-701e-0032-0ac8-56f9fd000000
X-Azure-Ref-OriginShield: Ref A: B79B1871E41B44099EC00B5BEA63D916 Ref B: BL2EDGE0112 Ref C: 2019-08-19T20:01:30Z
X-MSEdge-Ref: Ref A: D6E608C2C4E14A389233C4B794B487D3 Ref B: NYCEDGE1010 Ref C: 2019-08-19T20:01:30Z
Date: Mon, 19 Aug 2019 20:01:29 GMT

Another anomaly is that the version of the Google HTTP Client in the user agent is 1.30.0, which is supposed to be 1.31.0 that we use.

Google-HTTP-Java-Client/1.30.0 (gzip)

In any case, I suspect this is due to our upgrading the Google HTTP Client.

[chanseokoh]

Verified that it works after reverting #1882 (that upgrades Google HTTP Client to 1.31.0) to use the previous 1.27.0.

[chanseokoh]

It could be a bug in google-http-client 1.31.0. Filed googleapis/google-http-java-client#795. Let's see what they say. I'll keep digging in in the meantime.

[chanseokoh]

I suspect that google-http-client 1.31.0 has a bug and fails to capture and try the correct temporary redirect URL from the 307 response. The server is probably working correctly by returning "time-limited URL validation failure" for non-existing or incorrect URLs.

It is very unfortunate but I think we should revert back to 1.27.0 and be stuck with it for some time.

Also, from the example code in googleapis/google-http-java-client#795, it seems like a bug that the wrong version is reported. Filed googleapis/google-http-java-client#796.

[chanseokoh]

@m86194 Jib 1.5.1 is released and should now work with Microsoft ACR.