Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to install package aqua-proxy, but curl is succcessful. #3152

Open
davidjeddy opened this issue Oct 3, 2024 · 26 comments
Open

Unable to install package aqua-proxy, but curl is succcessful. #3152

davidjeddy opened this issue Oct 3, 2024 · 26 comments

Comments

@davidjeddy
Copy link

aqua info

[home]$ aqua info
{
  "version": "2.36.1",
  "commit_hash": "423bf97060599e911f9a5a2c5622cf886673dd65",
  "os": "linux",
  "arch": "amd64",
  "pwd": "/home/(USER)/toolchain-management",
  "root_dir": "/home/(USER)/.local/share/aquaproj-aqua",
  "env": {
    "AQUA_GLOBAL_CONFIG": "/home/(USER)/.aqua/aqua.yaml"
  },
  "config_files": [
    {
      "path": "/home/(USER)/toolchain/aqua.yaml"
    }
  ]
}
[home]$ aqua --log-level DEBUG install
ERRO[0000] install the registry                          aqua_version=2.36.1 env=linux/amd64 error="get a file by Get GitHub Content API: Get \"https://api.github.com/repos/aquaproj/aqua-registry/contents/?ref=v4.220.2\": read tcp 10.14.117.179:60188->4.208.26.200:443: read: connection reset by peer" program=aqua registry_name=standard
FATA[0000] aqua failed                                   aqua_version=2.36.1 env=linux/amd64 error="it failed to install some registries" program=aqua
[home]$ aqua --log-level DEBUG install
DEBU[0000] install the proxy                             aqua_version=2.36.1 env=linux/amd64 package_name=aqua-proxy package_version=v1.2.8 program=aqua registry=
DEBU[0000] check if aqua-proxy is already installed      aqua_version=2.36.1 env=linux/amd64 package_name=aqua-proxy package_version=v1.2.8 program=aqua registry=
DEBU[0000] check if the package is already installed     aqua_version=2.36.1 env=linux/amd64 package_name=aqua-proxy package_version=v1.2.8 program=aqua registry=
INFO[0000] download and unarchive the package            aqua_version=2.36.1 env=linux/amd64 package_name=aqua-proxy package_version=v1.2.8 program=aqua registry=
DEBU[0000] failed to download an asset from GitHub Release without GitHub API. Try again with GitHub API  aqua_version=2.36.1 asset_name=aqua-proxy_linux_amd64.tar.gz asset_version=v1.2.8 env=linux/amd64 error="send http request: Get \"https:/aquaproj/aqua-proxy/releases/download/v1.2.8/aqua-proxy_linux_amd64.tar.gz\": read tcp 10.14.117.179:60556->4.208.26.197:443: read: connection reset by peer" package_name=aqua-proxy package_version=v1.2.8 program=aqua registry= repo_name=aqua-proxy repo_owner=aquaproj
FATA[0000] aqua failed                                   aqua_version=2.36.1 env=linux/amd64 error="install aqua-proxy: get the GitHub Release by Tag: Get \"https://api.github.com/repos/aquaproj/aqua-proxy/releases/tags/v1.2.8\": read tcp 10.14.117.179:58092->4.208.26.200:443: read: connection reset by peer" program=aqua

However, if I use curl I am about to download the file.

[home]$ curl --location --output tmp.tar.gz https:/aquaproj/aqua-proxy/releases/download/v1.2.8/aqua-proxy_linux_amd64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  784k  100  784k    0     0  1374k      0 --:--:-- --:--:-- --:--:-- 85.1M
[home]$ ls -la
...
-rw-r--r--. 1 user user 803206 Oct  3 12:51 tmp.tar.gz
...

Just to be sure, I checked the ENV VAR list for a proxy config:

[home]$ printenv | sort
AQUA_GLOBAL_CONFIG=/home/user/.aqua/aqua.yaml
DEBUGINFOD_URLS=https://debuginfod.fedoraproject.org/ 
EDITOR=/usr/bin/nano
GOENV_ROOT=/home/user/.goenv
GOENV_SHELL=bash
GOPATH=/home/user/go/1.21.13
GOROOT=/home/user/.goenv/versions/1.21.13
HISTCONTROL=ignoredups
HISTSIZE=1000
HOME=/home/user
HOSTNAME=ip-10-14-117-179.eu-west-1.compute.internal
LANG=en_US.UTF-8
LESSOPEN=||/usr/bin/lesspipe.sh %s
LOGNAME=user
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;37;41:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.m4a=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.oga=01;36:*.opus=01;36:*.spx=01;36:*.xspf=01;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:
MAIL=/var/spool/mail/user
OLDPWD=/home/user
PATH=/home/user/.local/share/aquaproj-aqua/bin:/home/user/.local/share/aquaproj-aqua/bin:/usr/bin/sonar-scanner/bin:/usr/bin/maven/bin:/home/david/.local/bin:/home/user/.pyenv/plugins/pyenv-virtualenv/shims:/home/user/.pyenv/shims:/home/user/.pyenv/bin:/home/user/.goenv/bin:/home/user/.goenv/shims:/usr/bin/maven/bin:/home/david/.local/bin:/home/user/.pyenv/plugins/pyenv-virtualenv/shims:/home/user/.pyenv/bin:/home/user/.goenv/bin:/home/user/.goenv/shims:/home/david/.local/bin:/home/user/.pyenv/plugins/pyenv-virtualenv/shims:/home/user/.pyenv/bin:/home/user/.goenv/bin:/home/user/.goenv/shims:/home/user/.pyenv/plugins/pyenv-virtualenv/shims:/home/user/.pyenv/bin:/home/user/.goenv/bin:/home/user/.goenv/shims:/home/user/.goenv/bin:/home/user/.goenv/shims:/home/user/.local/share/aquaproj-aqua/bin:/home/user/.local/share/aquaproj-aqua/bin:/usr/bin/sonar-scanner/bin:/usr/bin/maven/bin:/home/david/.local/bin:/home/user/.pyenv/plugins/pyenv-virtualenv/shims:/home/user/.pyenv/bin:/home/user/.goenv/bin:/home/user/.goenv/shims:/usr/bin/maven/bin:/home/david/.local/bin:/home/user/.pyenv/plugins/pyenv-virtualenv/shims:/home/user/.pyenv/bin:/home/user/.goenv/bin:/home/user/.goenv/shims:/home/david/.local/bin:/home/user/.pyenv/plugins/pyenv-virtualenv/shims:/home/user/.pyenv/bin:/home/user/.goenv/bin:/home/user/.goenv/shims:/home/user/.pyenv/plugins/pyenv-virtualenv/shims:/home/user/.pyenv/bin:/home/user/.goenv/bin:/home/user/.goenv/shims:/home/user/.goenv/bin:/home/user/.goenv/shims:/home/user/.pyenv/plugins/pyenv-virtualenv/shims:/home/user/.pyenv/bin:/home/user/.goenv/bin:/home/user/.goenv/shims:/home/user/.local/bin:/home/user/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
PWD=/home/user/toolchain
PYENV_ROOT=/home/user/.pyenv
PYENV_SHELL=bash
PYENV_VIRTUALENV_INIT=1
SHELL=/bin/bash
SHLVL=1
TERM=xterm-256color
TF_PLUGIN_CACHE_DIR=/home/user/.terraform.d/plugin-cache/
USER=user
_=/usr/bin/printenv

Not sure why curl would be successful but the aqua binary fails.

Overview

Unable to install packages from behind egress DNS firewall. We must have the exact root DNS / hostname for all out going requests. We have added .github. but are still getting connection reset.

How to reproduce

aqua.yaml

checksum:
  enabled: true
  require_checksum: true
  supported_envs:
  - all
registries:
- type: standard
  ref: v4.220.2
packages:
- name: aquasecurity/[email protected]
- name: aquasecurity/[email protected]
- name: aws/[email protected]
- name: bridgecrewio/[email protected]
- name: Checkmarx/[email protected]
- name: flosell/[email protected]
- name: infracost/[email protected]
- name: jqlang/[email protected]
- name: mikefarah/[email protected]
- name: terraform-docs/[email protected]
- name: terraform-linters/[email protected]
- name: tfutils/[email protected]
- name: tgenv/[email protected]
- name: tofuutils/[email protected]
- name: xeol-io/[email protected]

Other related code such as local Registry


Executed command and output

$ aqua install

Debug output

$ 

Expected behaviour

Able to download pacakges.

Actual behaviour

connection reset

Note

No response

@davidjeddy davidjeddy added the bug Something isn't working label Oct 3, 2024
@suzuki-shunsuke
Copy link
Member

Unfortunately, I have no idea.

get a file by Get GitHub Content API:
Get \"https://api.github.com/repos/aquaproj/aqua-registry/contents/?ref=v4.220.2\":
read tcp 10.14.117.179:60188->4.208.26.200:443: read: connection reset by peer"

send http request: Get \"https:/aquaproj/aqua-proxy/releases/download/v1.2.8/aqua-proxy_linux_amd64.tar.gz\":
read tcp 10.14.117.179:60556->4.208.26.197:443:
read: connection reset by peer"

install aqua-proxy: get the GitHub Release by Tag:
Get \"https://api.github.com/repos/aquaproj/aqua-proxy/releases/tags/v1.2.8\":
read tcp 10.14.117.179:58092->4.208.26.200:443: read: connection reset by peer"

Seems like there was a network issue.
aqua simply calls HTTP requests and GitHub API, so I don't think this is a bug of aqua.

@suzuki-shunsuke suzuki-shunsuke removed the bug Something isn't working label Oct 3, 2024
@suzuki-shunsuke
Copy link
Member

  1. How often does the issue occur? Definitely? or sometimes?
  2. Can you reproduce the issue using other aqua versions such as v2.36.0 and v2.30.0?
  3. Can you reproduce the issue in other environments?

At least, aqua v2.36.1 works well in my laptop (macOS) and GitHub Actions (ubuntu-latest, macos-13, macos-14, windows-latest).

@davidjeddy
Copy link
Author

davidjeddy commented Oct 4, 2024

  1. How often does the issue occur? Definitely? or sometimes?
    Always

  2. Can you reproduce the issue using other aqua versions such as v2.36.0 and v2.30.0?
    No. Unable to install other version, same error

  3. Can you reproduce the issue in other environments?
    No. Machines outside the network work as expected

I know is a connectivity issue but do not understand why as curl is successful.

[[~]]$ aqua --log-level DEBUG install
DEBU[0000] install the proxy                             aqua_version=2.36.1 env=linux/amd64 package_name=aqua-proxy package_version=v1.2.8 program=aqua registry=
DEBU[0000] check if aqua-proxy is already installed      aqua_version=2.36.1 env=linux/amd64 package_name=aqua-proxy package_version=v1.2.8 program=aqua registry=
DEBU[0000] check if the package is already installed     aqua_version=2.36.1 env=linux/amd64 package_name=aqua-proxy package_version=v1.2.8 program=aqua registry=
INFO[0000] download and unarchive the package            aqua_version=2.36.1 env=linux/amd64 package_name=aqua-proxy package_version=v1.2.8 program=aqua registry=
DEBU[0000] failed to download an asset from GitHub Release without GitHub API. Try again with GitHub API  aqua_version=2.36.1 asset_name=aqua-proxy_linux_amd64.tar.gz asset_version=v1.2.8 env=linux/amd64 error="send http request: Get \"https:/aquaproj/aqua-proxy/releases/download/v1.2.8/aqua-proxy_linux_amd64.tar.gz\": read tcp 10.14.117.179:53942->4.208.26.197:443: read: connection reset by peer" package_name=aqua-proxy package_version=v1.2.8 program=aqua registry= repo_name=aqua-proxy repo_owner=aquaproj
FATA[0000] aqua failed                                   aqua_version=2.36.1 env=linux/amd64 error="install aqua-proxy: get the GitHub Release by Tag: Get \"https://api.github.com/repos/aquaproj/aqua-proxy/releases/tags/v1.2.8\": read tcp 10.14.117.179:35116->4.208.26.200:443: read: connection reset by peer" program=aqua
[[~]]$ curl --location --output aqua-proxy_linux_amd64.tar.gz https:/aquaproj/aqua-proxy/releases/download/v1.2.8/aqua-proxy_linux_amd64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  784k  100  784k    0     0  1201k      0 --:--:-- --:--:-- --:--:-- 54.7M
[[~]]$ ls -lah
total 1.7M
drwxr-xr-x. 1 jenkins jenkins  494 Oct  4 08:33 .
drwx------. 1 jenkins jenkins  448 Oct  4 08:21 ..
-rw-r--r--. 1 jenkins jenkins 785K Oct  4 08:33 aqua-proxy_linux_amd64.tar.gz
...

It would seem the aqua binary is not following HTTPS only redirect. Does aqua require both HTTP and HTTPS to follow 302 redirect?

@suzuki-shunsuke
Copy link
Member

We don't take care of redirect when downloading files by HTTP request for now.

http.DefaultClient is used.

ctrl, err := controller.InitializeInstallCommandController(c.Context, param, http.DefaultClient, r.Runtime)

func (dl *httpDownloader) Download(ctx context.Context, u string) (io.ReadCloser, int64, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
if err != nil {
return nil, 0, fmt.Errorf("create a http request: %w", err)
}
resp, err := dl.client.Do(req)
if err != nil {
return nil, 0, fmt.Errorf("send http request: %w", err)
}
if resp.StatusCode >= http.StatusBadRequest {
return resp.Body, 0, logerr.WithFields(errInvalidHTTPStatusCode, logrus.Fields{ //nolint:wrapcheck
"http_status_code": resp.StatusCode,
})
}
return resp.Body, resp.ContentLength, nil
}

@suzuki-shunsuke
Copy link
Member

📝 https://pkg.go.dev/net/http#Client

        // CheckRedirect specifies the policy for handling redirects.
	// If CheckRedirect is not nil, the client calls it before
	// following an HTTP redirect. The arguments req and via are
	// the upcoming request and the requests made already, oldest
	// first. If CheckRedirect returns an error, the Client's Get
	// method returns both the previous Response (with its Body
	// closed) and CheckRedirect's error (wrapped in a url.Error)
	// instead of issuing the Request req.
	// As a special case, if CheckRedirect returns ErrUseLastResponse,
	// then the most recent response is returned with its body
	// unclosed, along with a nil error.
	//
	// If CheckRedirect is nil, the Client uses its default policy,
	// which is to stop after 10 consecutive requests.
	CheckRedirect func(req *[Request](https://pkg.go.dev/net/http#Request), via []*[Request](https://pkg.go.dev/net/http#Request)) [error](https://pkg.go.dev/builtin#error)

@suzuki-shunsuke
Copy link
Member

It would seem the aqua binary is not following HTTPS only redirect. Does aqua require both HTTP and HTTPS to follow 302 redirect?

Sorry. I don't understand this well.

@davidjeddy
Copy link
Author

HTTP return code 302 is a redirect
website A -> website B

Does aqua follow HTTPS redirects or does it does it only follow HTTP redirects?

@suzuki-shunsuke
Copy link
Member

suzuki-shunsuke commented Oct 4, 2024

I think aqua follows HTTPS redirects.

I checked redirects using -v option.

curl -v --location --output aqua-proxy_linux_amd64.tar.gz https:/aquaproj/aqua-proxy/releases/download/v1.2.8/aqua-proxy_linux_amd64.tar.gz

https:/aquaproj/aqua-proxy/releases/download/v1.2.8/aqua-proxy_linux_amd64.tar.gz redirects to https://objects.githubusercontent.com/**.
aqua usually works well. I think this means aqua follows HTTPS redirect correctly.

@suzuki-shunsuke
Copy link
Member

In your environment, GitHub API doesn't work too.
I don't think the API needs redirects.

Get \"https://api.github.com/repos/aquaproj/aqua-proxy/releases/tags/v1.2.8\": read tcp 10.14.117.179:35116->4.208.26.200:443: read: connection reset by peer"

@davidjeddy
Copy link
Author

davidjeddy commented Oct 4, 2024

Indeed. However both curl ... and curl --location ... does work.

[~]$ curl --verbose https://api.github.com/repos/aquaproj/aqua-proxy/releases/tags/v1.2.8
* Host api.github.com:443 was resolved.
* IPv6: (none)
* IPv4: 4.208.26.200
*   Trying 4.208.26.200:443...
* Connected to api.github.com (4.208.26.200) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / x25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=*.github.com
*  start date: Mar  7 00:00:00 2024 GMT
*  expire date: Mar  7 23:59:59 2025 GMT
*  subjectAltName: host "api.github.com" matched cert's "*.github.com"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo ECC Domain Validation Secure Server CA
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256
*   Certificate level 1: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 2: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://api.github.com/repos/aquaproj/aqua-proxy/releases/tags/v1.2.8
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: api.github.com]
* [HTTP/2] [1] [:path: /repos/aquaproj/aqua-proxy/releases/tags/v1.2.8]
* [HTTP/2] [1] [user-agent: curl/8.6.0]
* [HTTP/2] [1] [accept: */*]
> GET /repos/aquaproj/aqua-proxy/releases/tags/v1.2.8 HTTP/2
> Host: api.github.com
> User-Agent: curl/8.6.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 200 
< date: Fri, 04 Oct 2024 12:06:57 GMT
< content-type: application/json; charset=utf-8
< cache-control: public, max-age=60, s-maxage=60
< vary: Accept,Accept-Encoding, Accept, X-Requested-With
< etag: W/"c9d6a709e3360549fb1d2c1711a32c3e7752d226588e3c2a8e0017c793c654e8"
< last-modified: Tue, 01 Oct 2024 23:38:22 GMT
< x-github-media-type: github.v3; format=json
< x-github-api-version-selected: 2022-11-28
< access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
< access-control-allow-origin: *
< strict-transport-security: max-age=31536000; includeSubdomains; preload
< x-frame-options: deny
< x-content-type-options: nosniff
< x-xss-protection: 0
< referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
< content-security-policy: default-src 'none'
< server: github.com
< x-ratelimit-limit: 60
< x-ratelimit-remaining: 58
< x-ratelimit-reset: 1728047109
< x-ratelimit-resource: core
< x-ratelimit-used: 2
< accept-ranges: bytes
< content-length: 20980
< x-github-request-id: 9E39:18CC71:61FB50:67C016:66FFDA61
< 
{
  "url": "https://api.github.com/repos/aquaproj/aqua-proxy/releases/177907024",
  "assets_url": "https://api.github.com/repos/aquaproj/aqua-proxy/releases/177907024/assets",
  "upload_url": "https://uploads.github.com/repos/aquaproj/aqua-proxy/releases/177907024/assets{?name,label}",
  "html_url": "https:/aquaproj/aqua-proxy/releases/tag/v1.2.8",
  "id": 177907024,
  "author": {
    "login": "github-actions[bot]",
    "id": 41898282,
    "node_id": "MDM6Qm90NDE4OTgyODI=",
    "avatar_url": "https://avatars.githubusercontent.com/in/15368?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/github-actions%5Bbot%5D",
    "html_url": "https:/apps/github-actions",
    "followers_url": "https://api.github.com/users/github-actions%5Bbot%5D/followers",
    "following_url": "https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}",
    "gists_url": "https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/github-actions%5Bbot%5D/subscriptions",
    "organizations_url": "https://api.github.com/users/github-actions%5Bbot%5D/orgs",
    "repos_url": "https://api.github.com/users/github-actions%5Bbot%5D/repos",
    "events_url": "https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}",
    "received_events_url": "https://api.github.com/users/github-actions%5Bbot%5D/received_events",
    "type": "Bot",
    "site_admin": false
  },
  "node_id": "RE_kwDOF9Swy84KmqVQ",
  "tag_name": "v1.2.8",
  "target_commitish": "main",
  "name": "v1.2.8",
  "draft": false,
  "prerelease": false,
  "created_at": "2024-10-01T23:27:33Z",
  "published_at": "2024-10-01T23:28:24Z",
  "assets": [
    ...
    {
      "url": "https://api.github.com/repos/aquaproj/aqua-proxy/releases/assets/196245903",
      "id": 196245903,
      "node_id": "RA_kwDOF9Swy84LsnmP",
      "name": "multiple.intoto.jsonl",
      "label": "",
      "uploader": {
        "login": "github-actions[bot]",
        "id": 41898282,
        "node_id": "MDM6Qm90NDE4OTgyODI=",
        "avatar_url": "https://avatars.githubusercontent.com/in/15368?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/github-actions%5Bbot%5D",
        "html_url": "https:/apps/github-actions",
        "followers_url": "https://api.github.com/users/github-actions%5Bbot%5D/followers",
        "following_url": "https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}",
        "gists_url": "https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/github-actions%5Bbot%5D/subscriptions",
        "organizations_url": "https://api.github.com/users/github-actions%5Bbot%5D/orgs",
        "repos_url": "https://api.github.com/users/github-actions%5Bbot%5D/repos",
        "events_url": "https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}",
        "received_events_url": "https://api.github.com/users/github-actions%5Bbot%5D/received_events",
        "type": "Bot",
        "site_admin": false
      },
      "content_type": "application/octet-stream",
      "state": "uploaded",
      "size": 15830,
      "download_count": 0,
      "created_at": "2024-10-01T23:29:18Z",
      "updated_at": "2024-10-01T23:29:19Z",
      "browser_download_url": "https:/aquaproj/aqua-proxy/releases/download/v1.2.8/multiple.intoto.jsonl"
    }
  ],
  "tarball_url": "https://api.github.com/repos/aquaproj/aqua-proxy/tarball/v1.2.8",
  "zipball_url": "https://api.github.com/repos/aquaproj/aqua-proxy/zipball/v1.2.8",
  "body": "[Pull Requests](https:/aquaproj/aqua-proxy/pulls?q=is%3Apr+milestone%3Av1.2.8) | [Issues](https:/aquaproj/aqua-proxy/issues?q=is%3Aissue+milestone%3Av1.2.8) | https:/aquaproj/aqua-proxy/compare/v1.2.7...v1.2.8\r\n\r\n## Update dependencies\r\n\r\nUpdate Go to 1.23.2\r\n\r\n## Create GitHub Artifact Attestations\r\n\r\n#592\r\n\r\nhttps:/aquaproj/aqua-proxy/attestations\r\n\r\n"
}
* Connection #0 to host api.github.com left intact

and w/ --location

[~]$ curl --location --verbose https://api.github.com/repos/aquaproj/aqua-proxy/releases/tags/v1.2.8
* Host api.github.com:443 was resolved.
* IPv6: (none)
* IPv4: 4.208.26.200
*   Trying 4.208.26.200:443...
* Connected to api.github.com (4.208.26.200) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / x25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=*.github.com
*  start date: Mar  7 00:00:00 2024 GMT
*  expire date: Mar  7 23:59:59 2025 GMT
*  subjectAltName: host "api.github.com" matched cert's "*.github.com"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo ECC Domain Validation Secure Server CA
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256
*   Certificate level 1: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 2: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://api.github.com/repos/aquaproj/aqua-proxy/releases/tags/v1.2.8
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: api.github.com]
* [HTTP/2] [1] [:path: /repos/aquaproj/aqua-proxy/releases/tags/v1.2.8]
* [HTTP/2] [1] [user-agent: curl/8.6.0]
* [HTTP/2] [1] [accept: */*]
> GET /repos/aquaproj/aqua-proxy/releases/tags/v1.2.8 HTTP/2
> Host: api.github.com
> User-Agent: curl/8.6.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 200 
< date: Fri, 04 Oct 2024 12:05:09 GMT
< content-type: application/json; charset=utf-8
< cache-control: public, max-age=60, s-maxage=60
< vary: Accept,Accept-Encoding, Accept, X-Requested-With
< etag: W/"e21a8bb6f42c5d6aaf9fa70c60f45c00e8b715f95624f765a28f8b32e98c8621"
< last-modified: Tue, 01 Oct 2024 23:38:22 GMT
< x-github-media-type: github.v3; format=json
< x-github-api-version-selected: 2022-11-28
< access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
< access-control-allow-origin: *
< strict-transport-security: max-age=31536000; includeSubdomains; preload
< x-frame-options: deny
< x-content-type-options: nosniff
< x-xss-protection: 0
< referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
< content-security-policy: default-src 'none'
< server: github.com
< x-ratelimit-limit: 60
< x-ratelimit-remaining: 59
< x-ratelimit-reset: 1728047109
< x-ratelimit-resource: core
< x-ratelimit-used: 1
< accept-ranges: bytes
< content-length: 20980
< x-github-request-id: 1861:9257A:44FCE3:493052:66FFD9F5
< 
{
  "url": "https://api.github.com/repos/aquaproj/aqua-proxy/releases/177907024",
  "assets_url": "https://api.github.com/repos/aquaproj/aqua-proxy/releases/177907024/assets",
  "upload_url": "https://uploads.github.com/repos/aquaproj/aqua-proxy/releases/177907024/assets{?name,label}",
  "html_url": "https:/aquaproj/aqua-proxy/releases/tag/v1.2.8",
  "id": 177907024,
  "author": {
    "login": "github-actions[bot]",
    "id": 41898282,
    "node_id": "MDM6Qm90NDE4OTgyODI=",
    "avatar_url": "https://avatars.githubusercontent.com/in/15368?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/github-actions%5Bbot%5D",
    "html_url": "https:/apps/github-actions",
    "followers_url": "https://api.github.com/users/github-actions%5Bbot%5D/followers",
    "following_url": "https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}",
    "gists_url": "https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/github-actions%5Bbot%5D/subscriptions",
    "organizations_url": "https://api.github.com/users/github-actions%5Bbot%5D/orgs",
    "repos_url": "https://api.github.com/users/github-actions%5Bbot%5D/repos",
    "events_url": "https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}",
    "received_events_url": "https://api.github.com/users/github-actions%5Bbot%5D/received_events",
    "type": "Bot",
    "site_admin": false
  },
  "node_id": "RE_kwDOF9Swy84KmqVQ",
  "tag_name": "v1.2.8",
  "target_commitish": "main",
  "name": "v1.2.8",
  "draft": false,
  "prerelease": false,
  "created_at": "2024-10-01T23:27:33Z",
  "published_at": "2024-10-01T23:28:24Z",
  "assets": [
    ...
    {
      "url": "https://api.github.com/repos/aquaproj/aqua-proxy/releases/assets/196245903",
      "id": 196245903,
      "node_id": "RA_kwDOF9Swy84LsnmP",
      "name": "multiple.intoto.jsonl",
      "label": "",
      "uploader": {
        "login": "github-actions[bot]",
        "id": 41898282,
        "node_id": "MDM6Qm90NDE4OTgyODI=",
        "avatar_url": "https://avatars.githubusercontent.com/in/15368?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/github-actions%5Bbot%5D",
        "html_url": "https:/apps/github-actions",
        "followers_url": "https://api.github.com/users/github-actions%5Bbot%5D/followers",
        "following_url": "https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}",
        "gists_url": "https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/github-actions%5Bbot%5D/subscriptions",
        "organizations_url": "https://api.github.com/users/github-actions%5Bbot%5D/orgs",
        "repos_url": "https://api.github.com/users/github-actions%5Bbot%5D/repos",
        "events_url": "https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}",
        "received_events_url": "https://api.github.com/users/github-actions%5Bbot%5D/received_events",
        "type": "Bot",
        "site_admin": false
      },
      "content_type": "application/octet-stream",
      "state": "uploaded",
      "size": 15830,
      "download_count": 0,
      "created_at": "2024-10-01T23:29:18Z",
      "updated_at": "2024-10-01T23:29:19Z",
      "browser_download_url": "https:/aquaproj/aqua-proxy/releases/download/v1.2.8/multiple.intoto.jsonl"
    }
  ],
  "tarball_url": "https://api.github.com/repos/aquaproj/aqua-proxy/tarball/v1.2.8",
  "zipball_url": "https://api.github.com/repos/aquaproj/aqua-proxy/zipball/v1.2.8",
  "body": "[Pull Requests](https:/aquaproj/aqua-proxy/pulls?q=is%3Apr+milestone%3Av1.2.8) | [Issues](https:/aquaproj/aqua-proxy/issues?q=is%3Aissue+milestone%3Av1.2.8) | https:/aquaproj/aqua-proxy/compare/v1.2.7...v1.2.8\r\n\r\n## Update dependencies\r\n\r\nUpdate Go to 1.23.2\r\n\r\n## Create GitHub Artifact Attestations\r\n\r\n#592\r\n\r\nhttps:/aquaproj/aqua-proxy/attestations\r\n\r\n"
}
* Connection #0 to host api.github.com left intact

I am even able to telnet to both api.github.com and objects.githubusercontent.com.

[jenkins@ip-10-14-117-179 toolchain-management]$ telnet api.github.com 443
Trying 4.208.26.200...
Connected to api.github.com.
Escape character is '^]'.
Connection closed by foreign host.
[jenkins@ip-10-14-117-179 toolchain-management]$ telnet objects.githubusercontent.com 443
Trying 185.199.109.133...
Connected to objects.githubusercontent.com.
Escape character is '^]'.
Connection closed by foreign host.

@suzuki-shunsuke
Copy link
Member

Are you familiar with Go?
Can you run the following script in your environment?

main.go:

package main

import (
	"context"
	"fmt"
	"io"
	"log"
	"net/http"
)

func main() {
	if err := core(); err != nil {
		log.Fatal(err)
	}
}

func core() error {
	u := "https:/aquaproj/aqua-proxy/releases/download/v1.2.8/aqua-proxy_linux_amd64.tar.gz"
	ctx := context.Background()
	client := &http.Client{}
	req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
	if err != nil {
		return fmt.Errorf("create a http request: %w", err)
	}
	resp, err := client.Do(req)
	if err != nil {
		return fmt.Errorf("send http request: %w", err)
	}
	defer resp.Body.Close()
	b, err := io.ReadAll(resp.Body)
	if err != nil {
		return fmt.Errorf("read a response body: %w", err)
	}
	log.Printf("status code: %d\n", resp.StatusCode)
	if resp.StatusCode < 300 {
		log.Println("Success!")
	} else {
		log.Printf("body: %s", string(b))
	}
	return nil
}
go version
go run main.go

I expect you can reproduce the issue using this code.
Then we may be able to ask Go community for help.

@davidjeddy
Copy link
Author

I can get around with Go, it's been awhile.

$ go version

go1.21.13 linux/amd64

$ cd $HOME
$ mkdir test
$ vi test/main.go # add script to test/main.go, save, exit
$ go mod init test/main.go
$ cd test
$ go run .

2024/10/04 13:15:38 status code: 200
2024/10/04 13:15:38 Success!

@suzuki-shunsuke
Copy link
Member

Oh? Looks like the issue wasn't reproduced.
The above code is basically same with aqua.
Interesting.

@davidjeddy
Copy link
Author

I have reached out to my networking group with our findings. 🤞 Would be nice if the solution is a egress rule problem. Will close this issue as I believe it is not an aqua problem.

Thank you greatly for your help and fast response. This is an awesome project with an amazing maintainer.

@davidjeddy
Copy link
Author

Was able to verify with the network team it is indeed a firewall rule issue. Sorry for the bother, but again, thank you for the assistance. Keep up the great effort.

@davidjeddy davidjeddy reopened this Oct 11, 2024
@davidjeddy
Copy link
Author

Sorry to do this but having to re-open the issue. After troubleshooting w/ our network team this is our findings:

Disable firewall rules:

  • All successful

Enable firewall rules:

If it helps the firewall is the AWS Network firewall. It evaluates the requested domain on HTTP/TLS and allows traffic out only if the request matches the allow list.

The network team noticed that aqua is not handling the TLS cert the same way curl/go script does. Is the http package in aqua configured differently?

@suzuki-shunsuke
Copy link
Member

suzuki-shunsuke commented Oct 11, 2024

  1. GitHub API
  2. HTTP request

--

  1. About GitHub API, aqua creates http.Client using oauth2#NewClient.

func getHTTPClientForGitHub(ctx context.Context, token string) *http.Client {
if token == "" {
return http.DefaultClient
}
return oauth2.NewClient(ctx, oauth2.StaticTokenSource(
&oauth2.Token{AccessToken: token},
))
}

The different between this client and go script is that this client has the custom Transport.

https://cs.opensource.google/go/x/oauth2/+/refs/tags/v0.23.0:oauth2.go;l=360-363

	return &http.Client{
		Transport: &Transport{
			Base:   internal.ContextClient(ctx).Transport,
			Source: ReuseTokenSource(nil, src),
		},
	}
  1. About HTTP, aqua uses http.DefaultClient

ctrl, err := controller.InitializeInstallCommandController(c.Context, param, http.DefaultClient, r.Runtime)

http.DefaultClient is same with &http.Client{} unless it is modified in Go code.

https://pkg.go.dev/net/http#pkg-variables
https://cs.opensource.google/go/go/+/refs/tags/go1.23.2:src/net/http/client.go;l=109

I don't think aqua modifies http.DefaultClient.

@suzuki-shunsuke
Copy link
Member

Can you edit the go script and replace &http.Client{} with http.DefaultClient just in case?

#3152 (comment)

package main

import (
	"context"
	"fmt"
	"io"
	"log"
	"net/http"
)

func main() {
	if err := core(); err != nil {
		log.Fatal(err)
	}
}

func core() error {
	u := "https:/aquaproj/aqua-proxy/releases/download/v1.2.8/aqua-proxy_linux_amd64.tar.gz"
	ctx := context.Background()
	client := http.DefaultClient // CHANGED
	req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
	if err != nil {
		return fmt.Errorf("create a http request: %w", err)
	}
	resp, err := client.Do(req)
	if err != nil {
		return fmt.Errorf("send http request: %w", err)
	}
	defer resp.Body.Close()
	b, err := io.ReadAll(resp.Body)
	if err != nil {
		return fmt.Errorf("read a response body: %w", err)
	}
	log.Printf("status code: %d\n", resp.StatusCode)
	if resp.StatusCode < 300 {
		log.Println("Success!")
	} else {
		log.Printf("body: %s", string(b))
	}
	return nil
}

@suzuki-shunsuke
Copy link
Member

The network team noticed that aqua is not handling the TLS cert the same way curl/go script does.

I'm not familiar with Network, but could you explain more detail? What's the difference?

@suzuki-shunsuke
Copy link
Member

I'm not sure if this is related to this issue, but Go's net/http client reuses TCP connections.

https://pkg.go.dev/net/http#Client

The [Client.Transport] typically has internal state (cached TCP connections), so Clients should be reused instead of created as needed. Clients are safe for concurrent use by multiple goroutines.

@davidjeddy
Copy link
Author

The network team noticed that aqua is not handling the TLS cert the same way curl/go script does.

I'm not familiar with Network, but could you explain more detail? What's the difference?

It's an AWS service that provides egress firewall protection using remote host names (DNS). To allow traffic out of a VPC network the remote host name must match an allow list. Else the traffic is rejected (connect reset) or dropped.

https://aws.amazon.com/network-firewall/

I'm not sure if this is related to this issue, but Go's net/http client reuses TCP connections.

pkg.go.dev/net/http#Client

The [Client.Transport] typically has internal state (cached TCP connections), so Clients should be reused instead of created as needed. Clients are safe for concurrent use by multiple goroutines.

I am not sure of the impact regarding this. :/

[~]$ which go
~/.goenv/shims/go
[~]$ go version
go version go1.21.13 linux/amd64
[~]$ mkdir test
[~]$ cd t
-bash: cd: t: No such file or directory
[~]$ cd test/
[test]$ vi test.go
[test]$ go mod init test/test
go: creating new go.mod: module test/test
go: to add module requirements and sums:
	go mod tidy
[test]$ go mod tidy
[test]$ go run test.go 
2024/10/15 08:04:19 status code: 200
2024/10/15 08:04:19 Success!

However when aqua runs:

[test]$ aqua --log-level TRACE install
DEBU[0000] install the proxy                             aqua_version=2.36.1 env=linux/amd64 package_name=aqua-proxy package_version=v1.2.8 program=aqua registry=
DEBU[0000] check if aqua-proxy is already installed      aqua_version=2.36.1 env=linux/amd64 package_name=aqua-proxy package_version=v1.2.8 program=aqua registry=
DEBU[0000] check if the package is already installed     aqua_version=2.36.1 env=linux/amd64 package_name=aqua-proxy package_version=v1.2.8 program=aqua registry=
INFO[0000] download and unarchive the package            aqua_version=2.36.1 env=linux/amd64 package_name=aqua-proxy package_version=v1.2.8 program=aqua registry=
DEBU[0000] failed to download an asset from GitHub Release without GitHub API. Try again with GitHub API  aqua_version=2.36.1 asset_name=aqua-proxy_linux_amd64.tar.gz asset_version=v1.2.8 env=linux/amd64 error="send http request: Get \"https:/aquaproj/aqua-proxy/releases/download/v1.2.8/aqua-proxy_linux_amd64.tar.gz\": read tcp 10.14.112.124:60978->4.208.26.197:443: read: connection reset by peer" package_name=aqua-proxy package_version=v1.2.8 program=aqua registry= repo_name=aqua-proxy repo_owner=aquaproj
FATA[0000] aqua failed                                   aqua_version=2.36.1 env=linux/amd64 error="install aqua-proxy: get the GitHub Release by Tag: Get \"https://api.github.com/repos/aquaproj/aqua-proxy/releases/tags/v1.2.8\": read tcp 10.14.112.124:52078->4.208.26.200:443: read: connection reset by peer" program=aqua

@suzuki-shunsuke
Copy link
Member

Sorry. Seems like I closed this issue mistakenly.

@davidjeddy
Copy link
Author

davidjeddy commented Oct 15, 2024

If it helps, here is the tcpdump from three attempted installed:

[~]$ sudo tcpdump -i eth0 host api.github.com
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
08:49:17.177134 IP [[REDACTED_SOURCE_HOST_NAME]].51172 > 4.208.26.200.https: Flags [S], seq 2320091743, win 62727, options [mss 8961,sackOK,TS val 2654682921 ecr 0,nop,wscale 7], length 0
08:49:17.189319 IP 4.208.26.200.https > [[REDACTED_SOURCE_HOST_NAME]].51172: Flags [S.], seq 4163111738, ack 2320091744, win 65535, options [mss 1436,sackOK,TS val 2781344 ecr 2654682921,nop,wscale 10], length 0
08:49:17.189333 IP [[REDACTED_SOURCE_HOST_NAME]].51172 > 4.208.26.200.https: Flags [.], ack 1, win 491, options [nop,nop,TS val 2654682933 ecr 2781344], length 0
08:49:17.189656 IP [[REDACTED_SOURCE_HOST_NAME]].51172 > 4.208.26.200.https: Flags [.], seq 1:1425, ack 1, win 491, options [nop,nop,TS val 2654682933 ecr 2781344], length 1424
08:49:17.189659 IP [[REDACTED_SOURCE_HOST_NAME]].51172 > 4.208.26.200.https: Flags [P.], seq 1425:1495, ack 1, win 491, options [nop,nop,TS val 2654682933 ecr 2781344], length 70
08:49:17.190141 IP 4.208.26.200.https > [[REDACTED_SOURCE_HOST_NAME]].51172: Flags [R.], seq 1, ack 1425, win 491, length 0
08:49:17.647967 IP [[REDACTED_SOURCE_HOST_NAME]].51186 > 4.208.26.200.https: Flags [S], seq 4113302268, win 62727, options [mss 8961,sackOK,TS val 2654683392 ecr 0,nop,wscale 7], length 0
08:49:17.652608 IP 4.208.26.200.https > [[REDACTED_SOURCE_HOST_NAME]].51186: Flags [S.], seq 867288128, ack 4113302269, win 65535, options [mss 1436,sackOK,TS val 1804392669 ecr 2654683392,nop,wscale 10], length 0
08:49:17.652625 IP [[REDACTED_SOURCE_HOST_NAME]].51186 > 4.208.26.200.https: Flags [.], ack 1, win 491, options [nop,nop,TS val 2654683396 ecr 1804392669], length 0
08:49:17.652953 IP [[REDACTED_SOURCE_HOST_NAME]].51186 > 4.208.26.200.https: Flags [.], seq 1:1425, ack 1, win 491, options [nop,nop,TS val 2654683397 ecr 1804392669], length 1424
08:49:17.652957 IP [[REDACTED_SOURCE_HOST_NAME]].51186 > 4.208.26.200.https: Flags [P.], seq 1425:1495, ack 1, win 491, options [nop,nop,TS val 2654683397 ecr 1804392669], length 70
08:49:17.653412 IP 4.208.26.200.https > [[REDACTED_SOURCE_HOST_NAME]].51186: Flags [R.], seq 1, ack 1425, win 491, length 0
08:49:18.048460 IP [[REDACTED_SOURCE_HOST_NAME]].51190 > 4.208.26.200.https: Flags [S], seq 546040360, win 62727, options [mss 8961,sackOK,TS val 2654683792 ecr 0,nop,wscale 7], length 0
08:49:18.056588 IP 4.208.26.200.https > [[REDACTED_SOURCE_HOST_NAME]].51190: Flags [S.], seq 2119551407, ack 546040361, win 65535, options [mss 1436,sackOK,TS val 1239452391 ecr 2654683792,nop,wscale 10], length 0
08:49:18.056606 IP [[REDACTED_SOURCE_HOST_NAME]].51190 > 4.208.26.200.https: Flags [.], ack 1, win 491, options [nop,nop,TS val 2654683800 ecr 1239452391], length 0
08:49:18.056958 IP [[REDACTED_SOURCE_HOST_NAME]].51190 > 4.208.26.200.https: Flags [.], seq 1:1425, ack 1, win 491, options [nop,nop,TS val 2654683801 ecr 1239452391], length 1424
08:49:18.056961 IP [[REDACTED_SOURCE_HOST_NAME]].51190 > 4.208.26.200.https: Flags [P.], seq 1425:1495, ack 1, win 491, options [nop,nop,TS val 2654683801 ecr 1239452391], length 70
08:49:18.057465 IP 4.208.26.200.https > [[REDACTED_SOURCE_HOST_NAME]].51190: Flags [R.], seq 1, ack 1425, win 491, length 0
  • Sorry I have to redact the source host name as to not give away our CIDR schema.

Status at this point:

  • Firewall off
    • curl: OK
    • Both Golang troubleshooting scripts: OK
    • Aqua: OK
  • Firewall on
    • curl: OK
    • Both Golang troubleshooting scripts: OK
    • Aqua: No

@davidjeddy
Copy link
Author

Did a little more troubleshooting. I thought I wonder if other package managers have this problem. I remember seeing asdf as an alternative to Aqua; so I gave that a try.

[~]$ asdf plugin add aqua-proxy https:/aquaproj/aqua-proxy.git
[~]$ asdf plugin list
aqua-proxy
nodejs
[~]$ asdf install aqua-proxy latest
Plugin aqua-proxy's list-all callback script failed with output:
/home/jenkins/.asdf/lib/functions/versions.bash: line 98: /home/jenkins/.asdf/plugins/aqua-proxy/bin/list-all: No such file or directory

aqua-proxy  is already installed
[~]$ asdf plugin list --urls
aqua-proxy                   https:/aquaproj/aqua-proxy.git
nodejs                       https:/asdf-vm/asdf-nodejs.git
[~]$ asdf plugin update --all
Location of nodejs plugin: /home/jenkins/.asdf/plugins/nodejs
Updating nodejs to master
Already on 'master'
Your branch is up to date with 'origin/master'.
Location of aqua-proxy plugin: /home/jenkins/.asdf/plugins/aqua-proxy
Updating aqua-proxy to main
Already on 'main'
Your branch is up to date with 'origin/main'.
[~]$ ls -la /home/jenkins/.asdf/plugins/aqua-proxy
total 56
drwxr-xr-x. 1 jenkins jenkins   306 Oct 15 09:00 .
drwxr-xr-x. 1 jenkins jenkins    32 Oct 15 09:00 ..
drwxr-xr-x. 1 jenkins jenkins   206 Oct 15 09:00 aqua
-rw-r--r--. 1 jenkins jenkins 10056 Oct 15 09:00 aqua-checksums.json
-rw-r--r--. 1 jenkins jenkins   252 Oct 15 09:00 aqua.yaml
drwxr-xr-x. 1 jenkins jenkins    20 Oct 15 09:00 cmd
-rw-r--r--. 1 jenkins jenkins  1322 Oct 15 09:00 cmdx.yaml
-rw-r--r--. 1 jenkins jenkins   113 Oct 15 09:00 CONTRIBUTING.md
drwxr-xr-x. 1 jenkins jenkins   158 Oct 15 09:01 .git
drwxr-xr-x. 1 jenkins jenkins    18 Oct 15 09:00 .github
-rw-r--r--. 1 jenkins jenkins    15 Oct 15 09:00 .gitignore
-rw-r--r--. 1 jenkins jenkins  1592 Oct 15 09:00 .golangci.yml
-rw-r--r--. 1 jenkins jenkins   147 Oct 15 09:00 go.mod
-rw-r--r--. 1 jenkins jenkins   846 Oct 15 09:00 .goreleaser.yml
-rw-r--r--. 1 jenkins jenkins   372 Oct 15 09:00 go.sum
-rw-r--r--. 1 jenkins jenkins  1072 Oct 15 09:00 LICENSE
drwxr-xr-x. 1 jenkins jenkins     6 Oct 15 09:00 pkg
-rw-r--r--. 1 jenkins jenkins   615 Oct 15 09:00 README.md
-rw-r--r--. 1 jenkins jenkins   326 Oct 15 09:00 renovate.json5
[~]$ cd /home/jenkins/.asdf/plugins/aqua-proxy/
[~]$ git remote -v
origin	https:/aquaproj/aqua-proxy.git (fetch)
origin	https:/aquaproj/aqua-proxy.git (push)

Git over HTTPS appears to be working in there implementation.

@suzuki-shunsuke
Copy link
Member

suzuki-shunsuke commented Oct 15, 2024

Thank you for your detail.

Can you run aqua init in the directory where aqua.yaml doesn't exist?
aqua init calls a GitHub API once and outputs warning if the API call fails.

release, _, err := c.github.GetLatestRelease(ctx, "aquaproj", "aqua-registry")
if err != nil {
logerr.WithError(logE, err).WithFields(logrus.Fields{
"repo_owner": "aquaproj",
"repo_name": "aqua-registry",
}).Warn("get the latest release")
} else {
if release == nil {
logE.WithFields(logrus.Fields{
"repo_owner": "aquaproj",
"repo_name": "aqua-registry",
}).Warn("failed to get the latest release")
} else {
registryVersion = release.GetTagName()
}
}

@suzuki-shunsuke
Copy link
Member

Can you check Firewall Alert log?

https://docs.aws.amazon.com/network-firewall/latest/developerguide/firewall-logging.html

I guess we can find the detail of blocked requests and why they were blocked.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants