When coredns fails traffic spiles out of k8s

[uk1988]

Hej All

Recently we have notices one thing and I would like to see if that is expected behavior or are we missing some configuration.
We are using default installation of rke2 with canal as CNI. Currently, for OS we are using RHEL 8.6.
As per default settings, subnet for pods is 10.42.x.x and subnet for services is 10.43.x.x.

In case core DNS in the cluster fails, outgoing pod connections start to look 10.43.x.x address outside k8s on the main LAN interface of the Linux nodes. In one case, this leads to network saturation outside k8s and crash of the network.
My question: Is this a bug or is it per design with calico that we must limit with Network policy unneeded egress traffic?

We have later seen that this can also happen let say in case pod running DB fails and pod calling the pod DB service starts to look out side of k8s network. Even when he is looking for services that are on address on 10.43.x.x

[brandond]

In case core DNS in the cluster fails, outgoing pod connections start to look 10.43.x.x address outside k8s on the main LAN interface of the Linux nodes.
We have later seen that this can also happen let say in case pod running DB fails and pod calling the pod DB service starts to look out side of k8s network. Even when he is looking for services that are on address on 10.43.x.x

Can you explain more about what specifically you're seeing? Are you just seeing DNS requests for cluster addresses hitting external DNS servers, or are you actually somehow seeing un-NATed traffic to pod or service addresses hitting the rest of your network?

[uk1988]

@brandond
Sorry for late replay was swapped with work...

For test I have one server which is both master and worker node for rke2.
Main network interface of linux VM is :

With tcpdump we monitor all traffic going from VM on main interface to the 10.43.0.0/16:
sudo tcpdump -ni eth0 'src net 10.25.16.0/24 and dst 10.43.0.0/16'

let say I kill rke2-coredns-rke2-coredns, tcpdump start to log following traffic on main interface:

When coredns recovers this DNS "spilling" stops.
Why is this the case? We have for fun set Network Policy to block all Egress trafic.
In such case there is no "spilling".

Furthermore, let say we have container A and container B.
Container B has a active TCP connection to container A.
If I restart container A then container B starts to look for TCP connection on main interface:

On one of our on prem test system QA left some pods in failed state over holidays.
The spilling request become so big that it saturated our local network considerably.

[brandond]

I would normally expect kube-router to black-hole the traffic if the service has no endpoints. It appears that instead of doing that, the traffic to the service ClusterIP is just leaving the CNI and going out to your local network. Have you customized the canal configuration at all?

[brandond]

cc @manuelbuil for additional consideration.

[uk1988]

We run default CNI settings, we have one NodePort that is for rabbitmq.
Here is the service view:

We did not test the same scenario on newer k8s version.

[brandond]

Have you modified anything else in your configuration? Are you adding any custom iptables rules on your host that might be interfering with the normal routing of traffic?

Here's what I see when I scale the coredns deployment down to zero:

root@rke2-server-1:/# kubectl scale -n kube-system deployment/rke2-coredns-rke2-coredns --replicas=0
deployment.apps/rke2-coredns-rke2-coredns scaled

root@rke2-server-1:/# iptables-save | grep 'KUBE-SERVICES.*coredns'
-A KUBE-SERVICES -d 10.43.0.10/32 -p udp -m comment --comment "kube-system/rke2-coredns-rke2-coredns:udp-53 has no endpoints" -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/rke2-coredns-rke2-coredns:tcp-53 has no endpoints" -j REJECT --reject-with icmp-port-unreachable

[manuelbuil]

Traffic to any service IP (default on 10.43.0.0/16) should be locally intercepted by iptables and destination IP gets replaced with the IP of a pod that implements that service (defaut on 10.42.0.0/16), i.e. a classical DNAT. As Brad is correctly explaining, if there are no pods implementing that service (BTW, this is called enpoints, and you can look at them by querying kubectl get endpoints -A), you should receive an icmp error that gets generated by iptables. Kube-proxy is the component that fills the iptables with the correct rules, so if you are missing the required iptables rules, there might be a problem with your kube-proxy

[uk1988]

No custom iptables rules. We are running vanilla RHEL 8.6 with with disabled firewalled service.
On top we are running rke2 v1.23.16+rke2r1.

We have both iptables and firewalled installed. Where firewalled is disabled as service:

Output of iptables rules:
iptables.txt

List of all packages:
installed_rpm.txt

[brandond]

That is a really old and end of life version of rke2. Can you reproduce this on any more recent release? 1.26+?

[uk1988]

k8s upgrade is long on the list, will try today with 1.28.x

[uk1988]

it was indeed old k8s, had no issues with latest 1.24 and 1.28 rke2 version

did not notice before but 1.23 was no supported/tested on RHEL 8.6
https://www.suse.com/suse-rke2/support-matrix/all-supported-versions/rke2-v1-23/