ACR authentication with token credentials is inconsistent (basic auth not accepted, but requested in 401 response)

[maxwolffe]

Hey all, first time ticket, love ACR in general, let me know if anything in here is unclear :)

Describe the bug
The tokens feature (in preview) seems to have inconsistent authentication semantics:
a. Token credentials provided in basic authentication fails but basic authentication is accepted with access key credentials.
b. Azure responds to failed authentication requests with an auth header WWW-Authenticate: Basic, but doesn't accept basic credentials (Bearer authentication is accepted. If basic auth is not supported for tokens, consider always responding with an allowed auth scheme.

This was initially reported to the jib project - GoogleContainerTools/jib#2784. They're making changes, but suggested that I also report it here, since there appears to be some oddness in ACR's authentication flow.

More details an debugging in the "Additional Context" section below.

Expected behavior
ACR accepts basic authentication for tokens or provides WWW-Authenticate: Bearer (or whichever is the preferred authentication method) headers in 401 responses.

Any relevant environment information

• Jib version: 2.5.2
• Build tool: Apache Maven 3.6.0
• OS: Ubuntu 18.04.4 LTS
• Datetime (UTC) when the issue occurred: 2020-09-24T14:34:42.4249224Z
• Registry and image names: mwolffetestingmicroservices.azurecr.io/base

Additional context
In long:
I'm running an AzureDevOps Pipelines build agent which is pulling a custom base image from Azure Container Registry, building an image, and pushing to the same registry. I'm using jib, a container packaging tool to pull a base image from my private ACR repo and then push the built image to the same repo. I'm supplying credentials to jib using the <to/from> command lines, like so:

/usr/bin/mvn -f /home/azureuser/myagent/_work/3/s/pom.xml --batch-mode -X \
  -Djava.util.logging.config.file=logging.properties \
  -Djib.serialize=true \
  -Djib.from.auth.username=*** -Djib.from.auth.password=*** \
  -Djib.to.auth.username=*** -Djib.to.auth.password=*** \
  -Djib.to.image=***testingmicroservices.azurecr.io/target-image:0.0.723 \
  jib:build

The to/from username and password are the same and are using a token which is configured to have both push and pull rights to all of my repositories (_repositories_admin has this property):

The pull authentication flow fails, because jib is first attempting basic authentication (ignoring the WWW-Authenticate: Bearer header - behavior they are changing), then is being instructed by Azure to attempt Basic authentication again (via the WWW-Authenticate header) which fails. This is the heart of the bug - Basic auth fails, but then basic auth is requested by ACR.

Bearer authentication does succeed (which is what jib is changing their flow to use)

debug log (click to expand)

2020-09-24T14:34:43.4524997Z [INFO] The base image requires auth. Trying again for ***testingmicroservices.azurecr.io/base@sha256:baseImageSha256Digest...
2020-09-24T14:34:43.4526527Z [INFO] Using credentials from <from><auth> for ***testingmicroservices.azurecr.io/base@sha256:baseImageSha256Digest
2020-09-24T14:34:43.4527698Z [DEBUG] Trying basic auth for ***testingmicroservices.azurecr.io/base@sha256:baseImageSha256Digest...
2020-09-24T14:34:43.4528766Z [DEBUG] configured basic auth for ***testingmicroservices.azurecr.io/base
2020-09-24T14:34:43.4542773Z Sep 24, 2020 2:34:43 PM com.google.api.client.http.HttpRequest execute
2020-09-24T14:34:43.4544316Z CONFIG: -------------- REQUEST  --------------
2020-09-24T14:34:43.4545930Z GET https://***testingmicroservices.azurecr.io/v2/base/manifests/sha256:baseImageSha256Digest
2020-09-24T14:34:43.4548092Z Accept: application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json
2020-09-24T14:34:43.4549605Z Accept-Encoding: gzip
2020-09-24T14:34:43.4551787Z Authorization: Basic XYZ
2020-09-24T14:34:43.4555096Z User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)
2020-09-24T14:34:43.4556172Z 
2020-09-24T14:34:43.4557045Z Sep 24, 2020 2:34:43 PM com.google.api.client.http.HttpRequest execute
2020-09-24T14:34:43.4561294Z CONFIG: curl -v --compressed -H 'Accept: application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json' -H 'Accept-Encoding: gzip' -H 'Authorization: Basic XYZ' -H 'User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)' -- 'https://***testingmicroservices.azurecr.io/v2/base/manifests/sha256:baseImageSha256Digest'
2020-09-24T14:34:43.5908561Z Sep 24, 2020 2:34:43 PM com.google.api.client.http.HttpResponse <init>
2020-09-24T14:34:43.5910623Z CONFIG: -------------- RESPONSE --------------
2020-09-24T14:34:43.5911370Z HTTP/1.1 401 Unauthorized
2020-09-24T14:34:43.5911934Z Server: openresty
2020-09-24T14:34:43.5912496Z Date: Thu, 24 Sep 2020 14:34:43 GMT
2020-09-24T14:34:43.5913547Z Content-Type: application/json; charset=utf-8
2020-09-24T14:34:43.5915335Z Content-Length: 198
2020-09-24T14:34:43.5916315Z Connection: keep-alive
2020-09-24T14:34:43.5917395Z Access-Control-Expose-Headers: Docker-Content-Digest
2020-09-24T14:34:43.5918595Z Access-Control-Expose-Headers: WWW-Authenticate
2020-09-24T14:34:43.5919945Z Access-Control-Expose-Headers: Link
2020-09-24T14:34:43.5921104Z Access-Control-Expose-Headers: X-Ms-Correlation-Request-Id
2020-09-24T14:34:43.5922428Z Docker-Distribution-Api-Version: registry/2.0
2020-09-24T14:34:43.5923747Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T14:34:43.5924828Z Www-Authenticate: Basic realm="Azure Container Registry"
2020-09-24T14:34:43.5925990Z X-Content-Type-Options: nosniff
2020-09-24T14:34:43.5927052Z X-Ms-Correlation-Request-Id: 7bf8fe87-09a7-4a01-8651-b64790a70eda
2020-09-24T14:34:43.5928363Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T14:34:43.5929244Z 
2020-09-24T14:34:43.5929873Z Sep 24, 2020 2:34:43 PM com.google.api.client.util.LoggingByteArrayOutputStream close
2020-09-24T14:34:43.5930610Z CONFIG: Total: 198 bytes
2020-09-24T14:34:43.5931280Z Sep 24, 2020 2:34:43 PM com.google.api.client.util.LoggingByteArrayOutputStream close
2020-09-24T14:34:43.5932531Z CONFIG: {"errors":[{"code":"UNAUTHORIZED","message":"authentication required, visit https://aka.ms/acr/authorization for more information.","detail":[{"Type":"repository","Name":"base","Action":"pull"}]}]}
2020-09-24T14:34:43.5933562Z 
2020-09-24T14:34:43.5934334Z Sep 24, 2020 2:34:43 PM com.google.api.client.http.HttpRequest execute
2020-09-24T14:34:43.5935421Z CONFIG: -------------- REQUEST  --------------
2020-09-24T14:34:43.5936250Z GET https://***testingmicroservices.azurecr.io/v2/
2020-09-24T14:34:43.5936824Z Accept: 
2020-09-24T14:34:43.5937852Z Accept-Encoding: gzip
2020-09-24T14:34:43.5939012Z Authorization: Basic XYZ
2020-09-24T14:34:43.5940642Z User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)
2020-09-24T14:34:43.5941485Z 
2020-09-24T14:34:43.5942126Z Sep 24, 2020 2:34:43 PM com.google.api.client.http.HttpRequest execute
2020-09-24T14:34:43.5944897Z CONFIG: curl -v --compressed -H 'Accept: ' -H 'Accept-Encoding: gzip' -H 'Authorization: Basic XYZ' -H 'User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)' -- 'https://***testingmicroservices.azurecr.io/v2/'
2020-09-24T14:34:43.5946644Z [DEBUG] Trying bearer auth for ***testingmicroservices.azurecr.io/base@sha256:baseImageSha256Digest...
2020-09-24T14:34:43.5947554Z [DEBUG] attempting bearer auth for ***testingmicroservices.azurecr.io/base...
2020-09-24T14:34:43.6540153Z Sep 24, 2020 2:34:43 PM com.google.api.client.http.HttpResponse <init>
2020-09-24T14:34:43.6542852Z CONFIG: -------------- RESPONSE --------------
2020-09-24T14:34:43.6544297Z HTTP/1.1 401 Unauthorized
2020-09-24T14:34:43.6545143Z Server: openresty
2020-09-24T14:34:43.6545941Z Date: Thu, 24 Sep 2020 14:34:43 GMT
2020-09-24T14:34:43.6547672Z Content-Type: application/json; charset=utf-8
2020-09-24T14:34:43.6551877Z Content-Length: 149
2020-09-24T14:34:43.6553468Z Connection: keep-alive
2020-09-24T14:34:43.6556159Z Access-Control-Expose-Headers: Docker-Content-Digest
2020-09-24T14:34:43.6557576Z Access-Control-Expose-Headers: WWW-Authenticate
2020-09-24T14:34:43.6558922Z Access-Control-Expose-Headers: Link
2020-09-24T14:34:43.6560310Z Access-Control-Expose-Headers: X-Ms-Correlation-Request-Id
2020-09-24T14:34:43.6563378Z Docker-Distribution-Api-Version: registry/2.0
2020-09-24T14:34:43.6565165Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T14:34:43.6566853Z Www-Authenticate: Basic realm="Azure Container Registry"
2020-09-24T14:34:43.6578265Z X-Content-Type-Options: nosniff
2020-09-24T14:34:43.6579734Z X-Ms-Correlation-Request-Id: c280b785-d194-47e3-bbdd-c7eb74eec80e
2020-09-24T14:34:43.6582435Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T14:34:43.6582855Z 
2020-09-24T14:34:43.6584240Z Sep 24, 2020 2:34:43 PM com.google.api.client.util.LoggingByteArrayOutputStream close
2020-09-24T14:34:43.6584827Z CONFIG: Total: 149 bytes
2020-09-24T14:34:43.6585385Z Sep 24, 2020 2:34:43 PM com.google.api.client.util.LoggingByteArrayOutputStream close
2020-09-24T14:34:43.6587170Z CONFIG: {"errors":[{"code":"UNAUTHORIZED","message":"authentication required, visit https://aka.ms/acr/authorization for more information.","detail":null}]}
2020-09-24T14:34:43.6588498Z 
2020-09-24T14:34:43.6589104Z [DEBUG] server requires basic auth for ***testingmicroservices.azurecr.io/base
2020-09-24T14:34:43.6590621Z [ERROR] The registry asked for basic authentication, but the registry had refused basic authentication previously
2020-09-24T14:34:43.6591329Z [DEBUG] TIMED	Pulling base image manifest : 251.0 ms
2020-09-24T14:34:43.6621287Z [DEBUG] TIMING	Preparing application layer builders
2020-09-24T14:34:43.6631007Z [DEBUG] TIMED	Preparing application layer builders : 1.0 ms
2020-09-24T14:34:43.6641263Z [INFO] Building dependencies layer...
2020-09-24T14:34:43.6644371Z [DEBUG] TIMING	Building dependencies layer
2020-09-24T14:34:45.0809577Z [DEBUG] Building dependencies layer built sha256:targetImageSha256
2020-09-24T14:34:45.0826552Z [DEBUG] TIMED	Building dependencies layer : 1418.0 ms
2020-09-24T14:34:45.0840672Z [INFO] Building resources layer...
2020-09-24T14:34:45.0848212Z [DEBUG] TIMING	Building resources layer
2020-09-24T14:34:45.1048175Z [DEBUG] Building resources layer built sha256:1d0b9a031e93e686a32f3f60566cfd02f9e52a0e8fed13c4ecfaa3037d4054cf
2020-09-24T14:34:45.1053357Z [DEBUG] TIMED	Building resources layer : 21.0 ms
2020-09-24T14:34:45.1071042Z [INFO] Building classes layer...
2020-09-24T14:34:45.1083081Z [DEBUG] TIMING	Building classes layer
2020-09-24T14:34:45.1286480Z [DEBUG] Building classes layer built sha256:2b22dea88866add0448162295beab663ce4cdb41d7a72602bb6018355c6c5318
2020-09-24T14:34:45.1291631Z [DEBUG] TIMED	Building classes layer : 21.0 ms
2020-09-24T14:34:45.1319166Z [DEBUG] TIMING	Preparing layer pushers
2020-09-24T14:34:45.1330904Z [DEBUG] TIMED	Preparing layer pushers : 1.0 ms
2020-09-24T14:34:45.1345258Z [DEBUG] TIMING	Pushing BLOB digest: sha256:targetImageSha256, size: 17016695
2020-09-24T14:34:45.1390901Z Sep 24, 2020 2:34:45 PM com.google.api.client.http.HttpRequest execute
2020-09-24T14:34:45.1395520Z CONFIG: -------------- REQUEST  --------------
2020-09-24T14:34:45.1411105Z HEAD https://***testingmicroservices.azurecr.io/v2/target-image/blobs/sha256:targetImageSha256
2020-09-24T14:34:45.1416219Z Accept: 
2020-09-24T14:34:45.1421939Z Accept-Encoding: gzip
2020-09-24T14:34:45.1463269Z Authorization: Bearer accessTokenResponse1
2020-09-24T14:34:45.1467837Z User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)
2020-09-24T14:34:45.1474595Z 
2020-09-24T14:34:45.1482604Z Sep 24, 2020 2:34:45 PM com.google.api.client.http.HttpRequest execute
2020-09-24T14:34:45.1521321Z CONFIG: curl -v --compressed -X HEAD -H 'Accept: ' -H 'Accept-Encoding: gzip' -H 'Authorization: Bearer accessTokenResponse1' -H 'User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)' -- 'https://***testingmicroservices.azurecr.io/v2/target-image/blobs/sha256:targetImageSha256'
2020-09-24T14:34:45.3842225Z Sep 24, 2020 2:34:45 PM com.google.api.client.http.HttpResponse <init>
2020-09-24T14:34:45.3843977Z CONFIG: -------------- RESPONSE --------------
2020-09-24T14:34:45.3844649Z HTTP/1.1 404 Not Found
2020-09-24T14:34:45.3845014Z Server: openresty
2020-09-24T14:34:45.3845460Z Date: Thu, 24 Sep 2020 14:34:45 GMT
2020-09-24T14:34:45.3846397Z Content-Type: application/json; charset=utf-8
2020-09-24T14:34:45.3847085Z Content-Length: 157
2020-09-24T14:34:45.3847740Z Connection: keep-alive
2020-09-24T14:34:45.3848860Z Access-Control-Expose-Headers: Docker-Content-Digest
2020-09-24T14:34:45.3849725Z Access-Control-Expose-Headers: WWW-Authenticate
2020-09-24T14:34:45.3850856Z Access-Control-Expose-Headers: Link
2020-09-24T14:34:45.3851723Z Access-Control-Expose-Headers: X-Ms-Correlation-Request-Id
2020-09-24T14:34:45.3852431Z Cache-Control: no-cache
2020-09-24T14:34:45.3853198Z Docker-Distribution-Api-Version: registry/2.0
2020-09-24T14:34:45.3854053Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T14:34:45.3855023Z X-Content-Type-Options: nosniff
2020-09-24T14:34:45.3855685Z X-Ms-Client-Request-Id: 
2020-09-24T14:34:45.3856533Z X-Ms-Correlation-Request-Id: bd322c68-58e7-4218-bb14-bf350d32682b
2020-09-24T14:34:45.3857330Z X-Ms-Request-Id: 6ccf72f7-a28e-4836-9ed4-e20d10fb3178
2020-09-24T14:34:45.3859306Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T14:34:45.3859643Z 
2020-09-24T14:34:45.4162266Z [DEBUG] TIMING	pushBlob
2020-09-24T14:34:45.4162975Z [DEBUG] 	TIMING	pushBlob POST sha256:targetImageSha256
2020-09-24T14:34:45.4213689Z Sep 24, 2020 2:34:45 PM com.google.api.client.http.HttpRequest execute
2020-09-24T14:34:45.4215668Z CONFIG: -------------- REQUEST  --------------
2020-09-24T14:34:45.4217386Z POST https://***testingmicroservices.azurecr.io/v2/target-image/blobs/uploads/?mount=sha256:targetImageSha256&from=base
2020-09-24T14:34:45.4218085Z Accept: 
2020-09-24T14:34:45.4219362Z Accept-Encoding: gzip
2020-09-24T14:34:45.4224707Z Authorization: Bearer accessTokenResponse1
2020-09-24T14:34:45.4228712Z User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)
2020-09-24T14:34:45.4229129Z 
2020-09-24T14:34:45.4229520Z Sep 24, 2020 2:34:45 PM com.google.api.client.http.HttpRequest execute
2020-09-24T14:34:45.4250394Z CONFIG: curl -v --compressed -X POST -H 'Accept: ' -H 'Accept-Encoding: gzip' -H 'Authorization: Bearer accessTokenResponse1' -H 'User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)' -- 'https://***testingmicroservices.azurecr.io/v2/target-image/blobs/uploads/?mount=sha256:targetImageSha256&from=base'
2020-09-24T14:34:45.6467158Z Sep 24, 2020 2:34:45 PM com.google.api.client.http.HttpResponse <init>
2020-09-24T14:34:45.6468915Z CONFIG: -------------- RESPONSE --------------
2020-09-24T14:34:45.6469382Z HTTP/1.1 202 Accepted
2020-09-24T14:34:45.6469901Z Server: openresty
2020-09-24T14:34:45.6470435Z Date: Thu, 24 Sep 2020 14:34:45 GMT
2020-09-24T14:34:45.6471133Z Content-Length: 0
2020-09-24T14:34:45.6471704Z Connection: keep-alive
2020-09-24T14:34:45.6472498Z Access-Control-Expose-Headers: Docker-Content-Digest
2020-09-24T14:34:45.6473325Z Access-Control-Expose-Headers: WWW-Authenticate
2020-09-24T14:34:45.6475505Z Access-Control-Expose-Headers: Link
2020-09-24T14:34:45.6476521Z Access-Control-Expose-Headers: X-Ms-Correlation-Request-Id
2020-09-24T14:34:45.6477411Z Docker-Distribution-Api-Version: registry/2.0
2020-09-24T14:34:45.6478286Z Docker-Upload-Uuid: dockerUploadId
2020-09-24T14:34:45.6480219Z Location: /v2/target-image/blobs/uploads/dockerUploadId?_nouploadcache=false&_state=9Zz4cun1jV2G6HgfuWfSwPuhjJcF_l2fjoJFiTRypoR7Ik5hbWUiOiJpbmJveC1zZXJ2aWNlIiwiVVVJRCI6ImY3MzljNzA1LTdhN2MtNDYzZC05OTU4LTNmYzljMTgyZjNlNSIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMC0wOS0yNFQxNDozNDo0NS42Mjc5NzIyMzJaIn0%3D
2020-09-24T14:34:45.6481569Z Range: 0-0
2020-09-24T14:34:45.6482366Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T14:34:45.6483171Z X-Content-Type-Options: nosniff
2020-09-24T14:34:45.6483861Z X-Ms-Client-Request-Id: 
2020-09-24T14:34:45.6484687Z X-Ms-Correlation-Request-Id: 8af63e2f-fd51-40c2-8c69-62f4cf2827ba
2020-09-24T14:34:45.6485670Z X-Ms-Request-Id: b75261dc-d8f4-45d3-a6a7-fd25a72f8556
2020-09-24T14:34:45.6486745Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T14:34:45.6487066Z 
2020-09-24T14:34:45.6487754Z [DEBUG] 	TIMED	pushBlob PATCH sha256:targetImageSha256 : 231.0 ms

However, if use an admin user access key, as below, the basic authentication succeeds.

debug log (click to expand)

2020-09-24T23:04:16.7850745Z [DEBUG] TIMING	Pulling base image manifest
2020-09-24T23:04:16.7924580Z Sep 24, 2020 11:04:16 PM com.google.api.client.http.HttpRequest execute
2020-09-24T23:04:16.7928417Z CONFIG: -------------- REQUEST  --------------
2020-09-24T23:04:16.7932745Z GET https://***.azurecr.io/v2/base/manifests/sha256:baseShaDigest
2020-09-24T23:04:16.7937465Z Accept: application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json
2020-09-24T23:04:16.7942063Z Accept-Encoding: gzip
2020-09-24T23:04:16.7945590Z User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)
2020-09-24T23:04:16.7948534Z 
2020-09-24T23:04:16.7955143Z Sep 24, 2020 11:04:16 PM com.google.api.client.http.HttpRequest execute
2020-09-24T23:04:16.7963014Z CONFIG: curl -v --compressed -H 'Accept: application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json' -H 'Accept-Encoding: gzip' -H 'User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)' -- 'https://***.azurecr.io/v2/base/manifests/sha256:baseShaDigest'
2020-09-24T23:04:16.8343471Z Sep 24, 2020 11:04:16 PM com.google.api.client.http.HttpResponse <init>
2020-09-24T23:04:16.8352555Z CONFIG: -------------- RESPONSE --------------
2020-09-24T23:04:16.8356605Z HTTP/1.1 401 Unauthorized
2020-09-24T23:04:16.8360129Z Server: openresty
2020-09-24T23:04:16.8363072Z Date: Thu, 24 Sep 2020 23:04:16 GMT
2020-09-24T23:04:16.8366799Z Content-Type: application/json; charset=utf-8
2020-09-24T23:04:16.8371728Z Content-Length: 198
2020-09-24T23:04:16.8375300Z Connection: keep-alive
2020-09-24T23:04:16.8382168Z Access-Control-Expose-Headers: Docker-Content-Digest
2020-09-24T23:04:16.8388775Z Access-Control-Expose-Headers: WWW-Authenticate
2020-09-24T23:04:16.8394608Z Access-Control-Expose-Headers: Link
2020-09-24T23:04:16.8401003Z Access-Control-Expose-Headers: X-Ms-Correlation-Request-Id
2020-09-24T23:04:16.8406830Z Docker-Distribution-Api-Version: registry/2.0
2020-09-24T23:04:16.8412737Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T23:04:16.8420887Z Www-Authenticate: Bearer realm="https://***.azurecr.io/oauth2/token",service="***.azurecr.io",scope="repository:base:pull"
2020-09-24T23:04:16.8425849Z X-Content-Type-Options: nosniff
2020-09-24T23:04:16.8430584Z X-Ms-Correlation-Request-Id: 2e14f07b-c536-4bb0-928c-aaa6cb68bd8a
2020-09-24T23:04:16.8439119Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T23:04:16.8443746Z 
2020-09-24T23:04:16.8464427Z Sep 24, 2020 11:04:16 PM com.google.api.client.util.LoggingByteArrayOutputStream close
2020-09-24T23:04:16.8468879Z CONFIG: Total: 198 bytes
2020-09-24T23:04:16.8482056Z Sep 24, 2020 11:04:16 PM com.google.api.client.util.LoggingByteArrayOutputStream close
2020-09-24T23:04:16.8489026Z CONFIG: {"errors":[{"code":"UNAUTHORIZED","message":"authentication required, visit https://aka.ms/acr/authorization for more information.","detail":[{"Type":"repository","Name":"base","Action":"pull"}]}]}
2020-09-24T23:04:16.8512154Z 
2020-09-24T23:04:16.8524562Z [INFO] The base image requires auth. Trying again for ***.azurecr.io/base@sha256:baseShaDigest...
2020-09-24T23:04:16.8536165Z [INFO] Using credentials from <from><auth> for ***.azurecr.io/base@sha256:baseShaDigest
2020-09-24T23:04:16.8543247Z [DEBUG] Trying basic auth for ***.azurecr.io/base@sha256:baseShaDigest...
2020-09-24T23:04:16.8555720Z [DEBUG] configured basic auth for ***.azurecr.io/base
2020-09-24T23:04:16.8582821Z Sep 24, 2020 11:04:16 PM com.google.api.client.http.HttpRequest execute
2020-09-24T23:04:16.8589098Z CONFIG: -------------- REQUEST  --------------
2020-09-24T23:04:16.8598063Z GET https://***.azurecr.io/v2/base/manifests/sha256:baseShaDigest
2020-09-24T23:04:16.8608805Z Accept: application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json
2020-09-24T23:04:16.8615472Z Accept-Encoding: gzip
2020-09-24T23:04:16.8667589Z Authorization: Basic basicAuthToken
2020-09-24T23:04:16.8679422Z User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)
2020-09-24T23:04:16.8679768Z 
2020-09-24T23:04:16.8697965Z Sep 24, 2020 11:04:16 PM com.google.api.client.http.HttpRequest execute
2020-09-24T23:04:16.8712890Z CONFIG: curl -v --compressed -H 'Accept: application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json' -H 'Accept-Encoding: gzip' -H 'Authorization: Basic basicAuthToken' -H 'User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)' -- 'https://***.azurecr.io/v2/base/manifests/sha256:baseShaDigest'
2020-09-24T23:04:17.1586567Z Sep 24, 2020 11:04:17 PM com.google.api.client.http.HttpResponse <init>
2020-09-24T23:04:17.1622014Z CONFIG: -------------- RESPONSE --------------
2020-09-24T23:04:17.1622645Z HTTP/1.1 200 OK
2020-09-24T23:04:17.1622931Z Server: openresty
2020-09-24T23:04:17.1623269Z Date: Thu, 24 Sep 2020 23:04:17 GMT
2020-09-24T23:04:17.1624208Z Content-Type: application/vnd.docker.distribution.manifest.v2+json
2020-09-24T23:04:17.1624950Z Content-Length: 1369
2020-09-24T23:04:17.1625610Z Connection: keep-alive
2020-09-24T23:04:17.1626329Z Access-Control-Expose-Headers: Docker-Content-Digest
2020-09-24T23:04:17.1629804Z Access-Control-Expose-Headers: WWW-Authenticate
2020-09-24T23:04:17.1630795Z Access-Control-Expose-Headers: Link
2020-09-24T23:04:17.1632509Z Access-Control-Expose-Headers: X-Ms-Correlation-Request-Id
2020-09-24T23:04:17.1633573Z Docker-Content-Digest: sha256:baseShaDigest
2020-09-24T23:04:17.1634481Z Docker-Distribution-Api-Version: registry/2.0
2020-09-24T23:04:17.1635597Z Etag: "sha256:baseShaDigest"
2020-09-24T23:04:17.1643580Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T23:04:17.1644598Z X-Content-Type-Options: nosniff
2020-09-24T23:04:17.1645292Z X-Ms-Client-Request-Id: 
2020-09-24T23:04:17.1646122Z X-Ms-Correlation-Request-Id: 042878ca-eaa5-4f7e-9904-928bff0b8789
2020-09-24T23:04:17.1647382Z X-Ms-Request-Id: c669691c-ea92-4bd9-852b-4c1763eeee39
2020-09-24T23:04:17.1648277Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T23:04:17.1648601Z 
2020-09-24T23:04:17.1973732Z Sep 24, 2020 11:04:17 PM com.google.api.client.util.LoggingByteArrayOutputStream close
2020-09-24T23:04:17.1974336Z CONFIG: Total: 1,369 bytes
2020-09-24T23:04:17.1974993Z Sep 24, 2020 11:04:17 PM com.google.api.client.util.LoggingByteArrayOutputStream close
2020-09-24T23:04:17.1975463Z CONFIG: {
2020-09-24T23:04:17.1975753Z    "schemaVersion": 2,
2020-09-24T23:04:17.1976197Z    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
2020-09-24T23:04:17.1976951Z    "config": {
2020-09-24T23:04:17.1977396Z       "mediaType": "application/vnd.docker.container.image.v1+json",
2020-09-24T23:04:17.1977841Z       "size": 4065,
2020-09-24T23:04:17.1978465Z       "digest": "sha256:otherDigest"
2020-09-24T23:04:17.1979065Z    },
2020-09-24T23:04:17.1979339Z    "layers": [
2020-09-24T23:04:17.1991074Z    ]
2020-09-24T23:04:17.1991308Z }

[yugangw-msft]

The repository based token is supposed to be consumed through basic auth. Protocol wise it is same like admin user account. @maxwolffe, want to confirm the registry name is testingmicroservices, and I will cross check the server log to see the root cause

[maxwolffe]

@yugangw-msft - Thanks for the response.

GoogleContainerTools/jib#2784 has been resolved, they changed their auth flow to use bearer auth when the first response from Azure returned WWW-Authenticate: Bearer and authentication with tokens succeeded.

The registry name is mwolffetestingmicroservices.azurecr.io.

I could probably recreate this issue for you using Jib 2.5.0 or earlier if it'd be helpful.

[yugangw-msft]

@maxwolffe, thanks for the information

@shizhMSFT, on repo based token, how does it work with basic auth? On this error, it appears we treated the repo-token as the service principal name which obviously won't work. I am looking at registry activities from mwolffetestingmicroservices between (datetime(2020-09-24 14:34:41) .. datetime(2020-09-24T14:34:45))

There is another issue happening earlier that through the bearer challenge, our token server issued the access token, but the subsequent request didn't use it for "Authorization" header, rather dropped it and went through the basic auth. This issue is not on us, and I guess that is what jib fixed.

[chanseokoh]

There is another issue happening earlier that through the bearer challenge, our token server issued the access token, but the subsequent request didn't use it for "Authorization" header, rather dropped it and went through the basic auth.

Looks like there's a bit of confusion, so allow me to clarify! Notice that Jib was interacting with two repositories: one for pulling and the other for pushing. Let's call them "base-repo" and "target-repo". Note that the "target-repo" is irrelevant of the failure in the auth flow we're talking about.

• "target-repo" for pushing: Jib did request a bearer token with both push and pull access. The auth server returned a token, which Jib successfully retrieved. Jib would subsequently use this token for accessing "target-repo". This token is not to be used for accessing "base-repo", so I'm emphasizing not using it is the correct behavior. No issue with the "target-repo" auth flow.
• "base-repo" for pulling: Jib never requested a bearer token. It only attempted passing basic auth, which failed.

Another issue is that, as @maxwolffe mentioned, right after the basic auth failed on "base-repo", ACR illogically asks Jib to do basic auth again.

[shizhMSFT]

@yugangw-msft The repository-scoped permissions (preview) feature is designed to be working only with bearer auth in the current stage.

By default, ACR challenges for bearer auth with Www-Authenticate: Bearer unless bearer auth is explicitly disabled via CSS support tickets. If the client requests with the header Authorization: Basic invalid_credential, ACR assumes the client can only accept basic auth and challenges with Www-Authenticate: Basic realm="Azure Container Registry" back.

[yugangw-msft]

@shizhMSFT, thanks for the clarification!

[yugangw-msft]

Closing as no follow up is required from ACR

[maxwolffe]

@yugangw-msft - is it not an issue that ACR is challenging with Basic auth (per @shizhMSFT's comment), but only accepts Bearer auth?

Or is it the case that tokens will support basic auth in the future?

[yugangw-msft]

Good question. CC @sajayantony for context why basic auth is not available with repo based permission token and whether we plan to add it in future.

[sajayantony]

There are multiple fundamental issue with Basic Auth and for ACR it's more about cost vs benefit of supporting multiple authentication mechanism. So from a direction standpoint we can assume that Bearer would most likely be the only supported model.