Using proxy for unix:// addresses???

[rubroboletus]

Hello,
I have just upgraded from CNI 1.5 to 1.6.1 on 1.14 EKS cluster and now I can see errors in imapd.log on worker node, after the debug message:
{"level":"debug","ts":"2020-04-28T07:30:50.835Z","caller":"ipamd/ipamd.go:476","msg":"Getting running pod sandboxes from "unix:///var/run/dockershim.sock""}

all the errors are from our PROXY server, with lines like:
{"level":"info","ts":"2020-04-28T07:30:50.837Z","caller":"ipamd/ipamd.go:387","msg":"Not able to get local pod sandboxes yet (attempt 5/5): rpc error: code = Unavailable desc = connection error:

redacted the rest, just our PROXY is stating that: "URL: CONNECT https://unix/
\\nCategory: Uncategorized URLs
\\nReason: UNKNOWN
\\nNotification: DNS_FAIL\\n

Same with:
{"level":"warn","ts":"2020-04-28T07:30:53.838Z","caller":"ipamd/ipamd.go:308"
{"level":"error","ts":"2020-04-28T07:30:53.838Z","caller":"aws-k8s-agent/main.go:30","msg":"Initialization failure: failed to get running pods!: Unable to get local pod sandboxes: rpc error:

Are you sure, you are handling a proxy settings for UNIX sockets correctly?

Regards,

Robert Hanzlik

[mogren]

Hi!

Did you use the config or just replace the image tag? The v1.6.x branch requires the dockershim.sock to be mounted. See these lines:

https:/aws/amazon-vpc-cni-k8s/blob/release-1.6.1/config/v1.6/aws-k8s-cni.yaml#L128-L145

Or, you can apply the full v1.6.1 config by doing:

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/release-1.6.1/config/v1.6/aws-k8s-cni.yaml

[rubroboletus]

Hi,

I don't think, that mounted / unmounted socket can affect using proxy for unix:// addresses. But installation was made by curl -OL https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/release-1.6.1/config/v1.6/aws-k8s-cni.yaml, modify image source to eu-central-1 region and adding proxy settings to environment part, then kubectl apply -f.

Regards,

Robi

[mogren]

Hi @rubroboletus! We have been trying to reproduce this issue without success so far. What was the proxy changes you were using?

Also, I wonder if this might be a related issue moby/moby#40817. What does docker version show on your worker nodes?

[rubroboletus]

Hi @mogren !, I was testing this on our "playground" EKS 1.14, which is in VPC with internal IP addresses only. Any internet access can be done via our proxy only. We have a very specific setup for this, including:

• set http_proxy + https_proxy + no_proxy (all lower and upper case) in /etc/environment file
• set the same in /etc/profile.d/proxy.sh
• set same proxy information in /etc/systemd/system.conf
• set same proxy information in /etc/systemd/system/docker.service.d/http-proxy.conf
  all running on Ubuntu.
  Also in ds deffinition we are adding in containers.env variables:
  HTTP_PROXY,HTTPS_PROXY,NO_PROXY
  kubectl describe node/node-name tells Container Runtime Version: docker://17.3.2

Regards,

Robi

[mogren]

@rubroboletus We are still investigating this. It seems it is a known gRPC issue that is still open. Some potential ways to solve this:

• Implement Fall back to using http docker client #752, and use that for Docker worker nodes
• Test the passthrough dialer option in cri.go.
• Long term: Fix Revise URI scheme parsing to be RFC 3986 compliant grpc/grpc-go#1911 upstream and use the fixed version

[nithu0115]

@mogren, I am able to replicate the issue on my side by setting up a squid proxy. I added NO_PROXY environment variable with /var/run/dockershim.sock, /var/run/docker.sock to aws-node daemonset, Kubelet, Kube-proxy, Docker daemon, and restarted Kubelet, Docker services to mitigate these errors.

@rubroboletus, Can you confirm if adding that environment var helps ?

[rubroboletus]

@nithu0115 was trying to reproduce your result with adding "/var/run...." to our NO_PROXY settings on worker nodes and daemonsets, but without success. In ipamd.log was same errors regarding using proxy for unix:/// with dockershim.sock.

[mogren]

A fix was done to gRPC by @pdbogen in grpc/grpc-go#3411, and it's in the latest release. Thanks @nithu0115 for finding it and opening #980. Until the next release is out, please use v1.5.7 to work around the issue.

[mogren]

Just referencing an old related ticket: #49

[mogren]

Should be fixed in the latest release, v1.6.2. @rubroboletus, feel free to verify, if possible.

[rubroboletus]

@mogren just tested the installation, seems be OK, will test it more deeply later today. Thanks for fixing.

[mogren]

Great! Thanks a lot for letting us know. 🙂