rke2 and konnectivity service

[rahulait]

Hi, I am trying to configure konnectivity service with RKE2. By default, I am seeing egress-selector-config.yaml configured for apiserver and it has following content:

{"kind":"EgressSelectorConfiguration","apiVersion":"apiserver.k8s.io/v1beta1","EgressSelections":[{"Name":"cluster","Connection":{"ProxyProtocol":"HTTPConnect","Transport":{"TCP":{"URL":"https://127.0.0.1:9345","TLSConfig":{"CABundle":"/var/lib/rancher/rke2/server/tls/server-ca.crt","ClientKey":"/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key","ClientCert":"/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt"}},"UDS":null}}}]}

If I try to change it, it gets auto-generated/overwritten whenever rke2-server.service is restarted.

In my setup, I am running control plane remotely (servers in one datacenter and workers in another). Worker nodes are able to join the cluster and pods are getting scheduled.

I am able to get konnectivity-server and konnectivity-agent installed on my rke2 cluster.

k get pods -n kube-system | grep konn
konnectivity-agent-4j8q2                               1/1     Running                 0                 48m
konnectivity-agent-7t5lk                               1/1     Running                 1 (29m ago)       4d20h
konnectivity-agent-gmb4l                               1/1     Running                 0                 51m
konnectivity-agent-t7kg7                               1/1     Running                 1 (33m ago)       4d20h
konnectivity-agent-xbjzl                               1/1     Running                 0                 4d20h
konnectivity-agent-znndp                               1/1     Running                 1 (24m ago)       4d20h
konnectivity-server-x5pq5um6wfrx3s2y5-master-0         1/1     Running                 0                 6d10h
konnectivity-server-x5pq5um6wfrx3s2y5-master-1         1/1     Running                 0                 6d10h
konnectivity-server-x5pq5um6wfrx3s2y5-master-2         1/1     Running                 0                 6d10h

I think for apiserver proxy, its using port 9345 instead of 8132 which I had configured for konnectivity-server. I am not sure if its coming because of the TCP URL set in egress-selector-config.yaml file.

curl http://127.0.0.1:8001/api/v1/namespaces/kube-system/services/nginx-service:80/proxy/
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "error trying to reach service: proxy error from 127.0.0.1:9345 while dialing 10.42.8.76:80, code 502: 502 Bad Gateway",
  "reason": "ServiceUnavailable",
  "code": 503
}

Am I reading it correct that its using 9345 instead of 8132 for konnectivity-service? Is there any documentation on how to setup konnectivity-service with rke2? Kindly do let me know.

[brandond]

RKE2 has a konektivity service equivalent built in to the main rke2 process. It is configured via the --egress-selector-mode option. The content covering this flag has not yet been migrated over to the RKE2 docs, you can reference the k3s docs for this feature at https://docs.k3s.io/installation/network-options#control-plane-egress-selector-configuration

[rahulait]

Thank you @brandond. I did used the flag but I might be doing something wrong due to which I am not seeing the correct behavior. Listed below is my setup.

RKE2 version: v1.28.3+rke2r1

I created 1 master node in region A and 1 worker node in region B. Each node has 2 ip-addresses on eth0: public ip (global) and private ip (local to region).

Config used to create master1 node (region A):

node-ip: 192.168.10.2
node-external-ip: 69.164.191.222
cni: cilium
disable: ['rke2-canal', 'rke2-ingress-nginx']
egress-selector-mode: agent

Config used to create worker1 node (region B):

node-ip: 192.168.11.5
node-external-ip: 173.255.200.10
server: https://69.164.191.222:9345
token: K101637295b959a4bf7ebea8d1d0e8<MODIFIED>f6d5ed::server:5ec071ec517<MODIFIED>e1386d5deb

I can see both the nodes ready:

k get nodes -o wide
NAME      STATUS   ROLES                       AGE   VERSION          INTERNAL-IP       EXTERNAL-IP      OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
master1   Ready    control-plane,etcd,master   92m   v1.28.3+rke2r1   192.168.10.2      69.164.191.222   Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.7.7-k3s1
worker1   Ready    <none>                      88m   v1.28.3+rke2r1   192.168.11.5      173.255.200.10   Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.7.7-k3s1

After this, I tried running nginx pod on worker1 and tried to access the service using api proxy, but its unable to route the traffic from apiserver to the nginx pod.

k get pods -n kube-system -o wide | grep nginx
nginx-deployment-68c9f4bbbc-bbhhw                      1/1     Running     0          58m   10.42.1.212       worker1   <none>           <none>

k get svc -n kube-system | grep nginx
nginx-service                      ClusterIP   10.43.96.136    <none>        80/TCP          58m

kubectl proxy --port=8080 &

curl http://127.0.0.1:8080/api/v1/namespaces/kube-system/services/nginx-service:80/proxy/
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "error trying to reach service: proxy error from 127.0.0.1:9345 while dialing 10.42.1.212:80, code 502: 502 Bad Gateway",
  "reason": "ServiceUnavailable",
  "code": 503
}

I tried different egress-selector modes but nothing has helped. I am not sure if its because I am using private node ip's and due to which cilium and vxlan are not happy. If I use public ip for node-ip in both master and worker, then it works. My assumption was that service like konnectivity allows control plane to reach worker nodes which might be behind a firewall by using forward proxy. However, if I use private ip for worker nodes, it fails to route the traffic to worker nodes. Hence confused as to where I might be thinking wrong.

Update:
I tried using canal instead of cilium and that works fine as expected even if I use private ip's for node ip. Not sure why cilium is not behaving the right way.

[brandond]

The konnectivity/egress stuff ONLY handles connections from the apiserver out to things running in the cluster. If you want pods (other than the apiserver) that are running on the server to be able to reach things hosted by pods running on the agents, then you need to ensure that your CNI works properly over the links between the two nodes.

Note that most CNIs use vxlan as their default transport, which is NOT secure and should not be run over the internet.

[brandond]

can you add debug: true to the config, restart rke2, and see what you get in the server logs when you try to proxy?

[rahulait]

Here are the logs:

Running pod:

k get pods -n kube-system -o wide | grep nginx
nginx-deployment-68c9f4bbbc-mgfjk                      1/1     Running     0          29h   10.42.235.130     worker1   <none>           <none>

On rke2-server:

Jan 24 21:20:04 master1 rke2[1454649]: time="2024-01-24T21:20:04Z" level=debug msg="Wrote ping"
Jan 24 21:20:04 master1 rke2[1454649]: time="2024-01-24T21:20:04Z" level=debug msg="Tunnel server handing HTTP/1.1 CONNECT request for //10.42.235.130:80 from 127.0.0.1:39204"
Jan 24 21:20:04 master1 rke2[1454649]: time="2024-01-24T21:20:04Z" level=debug msg="Tunnel server egress proxy dialing 10.42.235.130:80 directly"
Jan 24 21:20:09 master1 rke2[1454649]: time="2024-01-24T21:20:09Z" level=debug msg="Wrote ping"
Jan 24 21:20:14 master1 rke2[1454649]: time="2024-01-24T21:20:14Z" level=debug msg="Wrote ping"

On rke2-agent:

Jan 24 21:20:02 worker1 rke2[1090481]: time="2024-01-24T21:20:02Z" level=debug msg="Wrote ping"
Jan 24 21:20:07 worker1 rke2[1090481]: time="2024-01-24T21:20:07Z" level=debug msg="Wrote ping"
Jan 24 21:20:12 worker1 rke2[1090481]: time="2024-01-24T21:20:12Z" level=debug msg="Wrote ping"
Jan 24 21:20:17 worker1 rke2[1090481]: time="2024-01-24T21:20:17Z" level=debug msg="Wrote ping"

[brandond]

Are you restarting the worker too after changing the policy? You should see other log messages for pod addresses getting added and removed, as it tracks where to send the connections to.

[rahulait]

Hmm, I had restarted rke2-server.service and rke2-agent.service on master and worker respectively. As an additional precaution, I rebooted both my master and worker nodes and now I see even "kubectl proxy" working fine 😲. I'll try on a fresh setup and see once the cluster is up and running, does changing egress-selector-mode takes effect after service restart or only after node reboot. Thank you for helping with debugging till now, atleast I got some progress with it.

[brandond]

takes effect after a restart, but all cluster nodes need to be restarted to pick up the change.

[danielkza]

While trying out a bare-metal setup, I thought disabling the cloud controller was a good idea, and that led me to the same issue with Node IPs not being set.

Is it worth adding a small blurb in the documentation mentioning this is not recommended? Thanks!

[brandond]

We do warn about disabling things if you don't know what they're doing, but if there's some way we could warn about this more specifically, a PR would be welcome!