Node communication uses the wrong network interface? How to prevent that?

[kgts23]

Hello,

I am trying to build a cluster with 3 control planes and 4 agents. I am using Calico as the CNI. Everything was working great until I added a new network interface to the agents.

After adding a new network interface, I noticed that DNS resolution stopped working. On closer inspection, I found that the communication over VXLAN between the nodes is faulty. In tcpdump, I can see that the response on the control plane running the core DNS agent is going back through the newly added network interface instead of the correct one. I've tried to force the traffic flow with options like bind-address, node-ip & advertise-address, but the behavior remains the same.

NIC's on an agent

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether bc:24:11:a3:ae:e6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.21/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::be24:11ff:fea3:aee6/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UP group default qlen 1000
    link/ether bc:24:11:de:f2:93 brd ff:ff:ff:ff:ff:ff
    inet 172.16.7.71/24 brd 172.16.7.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::be24:11ff:fede:f293/64 scope link
       valid_lft forever preferred_lft forever
7: cali29a22595c7c@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-045a9ca6-7310-c8e2-6e59-7fcb04a8d06f
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever
8: cali550bd85d35b@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-cf58cd99-aae2-6d3a-00d7-ab8bc3bdd2ff
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever
9: vxlan.calico: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 66:6a:bc:6c:12:16 brd ff:ff:ff:ff:ff:ff
    inet 10.42.171.0/32 scope global vxlan.calico
       valid_lft forever preferred_lft forever
    inet6 fe80::646a:bcff:fe6c:1216/64 scope link
       valid_lft forever preferred_lft forever

default routing table on an agent

default via 192.168.1.1 dev eth0 proto static
10.42.37.192/26 via 10.42.37.192 dev vxlan.calico onlink
10.42.39.128/26 via 10.42.39.128 dev vxlan.calico onlink
10.42.48.0/26 via 10.42.48.0 dev vxlan.calico onlink
blackhole 10.42.171.0/26 proto 80
10.42.171.8 dev cali29a22595c7c scope link
10.42.171.9 dev cali550bd85d35b scope link
10.42.173.192/26 via 10.42.173.192 dev vxlan.calico onlink
10.42.182.192/26 via 10.42.182.192 dev vxlan.calico onlink
10.42.202.192/26 via 10.42.202.192 dev vxlan.calico onlink
172.16.7.0/24 dev eth1 proto kernel scope link src 172.16.7.71
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.21

all routing tables on an agent

ip route show table
Command line is not complete. Try option "help"
root@worker-01:~# ip route show table all
default via 192.168.1.1 dev eth0 proto static
10.42.37.192/26 via 10.42.37.192 dev vxlan.calico onlink
10.42.39.128/26 via 10.42.39.128 dev vxlan.calico onlink
10.42.48.0/26 via 10.42.48.0 dev vxlan.calico onlink
blackhole 10.42.171.0/26 proto 80
10.42.171.8 dev cali29a22595c7c scope link
10.42.171.9 dev cali550bd85d35b scope link
10.42.173.192/26 via 10.42.173.192 dev vxlan.calico onlink
10.42.182.192/26 via 10.42.182.192 dev vxlan.calico onlink
10.42.202.192/26 via 10.42.202.192 dev vxlan.calico onlink
172.16.7.0/24 dev eth1 proto kernel scope link src 172.16.7.71
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.21
local 10.42.171.0 dev vxlan.calico table local proto kernel scope host src 10.42.171.0
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
local 172.16.7.71 dev eth1 table local proto kernel scope host src 172.16.7.71
broadcast 172.16.7.255 dev eth1 table local proto kernel scope link src 172.16.7.71
local 192.168.1.21 dev eth0 table local proto kernel scope host src 192.168.1.21
broadcast 192.168.1.255 dev eth0 table local proto kernel scope link src 192.168.1.21
::1 dev lo proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev cali29a22595c7c proto kernel metric 256 pref medium
fe80::/64 dev cali550bd85d35b proto kernel metric 256 pref medium
fe80::/64 dev vxlan.calico proto kernel metric 256 pref medium
fe80::/64 dev eth1 proto kernel metric 256 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
local fe80::646a:bcff:fe6c:1216 dev vxlan.calico table local proto kernel metric 0 pref medium
local fe80::be24:11ff:fea3:aee6 dev eth0 table local proto kernel metric 0 pref medium
local fe80::be24:11ff:fede:f293 dev eth1 table local proto kernel metric 0 pref medium
local fe80::ecee:eeff:feee:eeee dev cali29a22595c7c table local proto kernel metric 0 pref medium
local fe80::ecee:eeff:feee:eeee dev cali550bd85d35b table local proto kernel metric 0 pref medium
multicast ff00::/8 dev eth0 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev cali29a22595c7c table local proto kernel metric 256 pref medium
multicast ff00::/8 dev cali550bd85d35b table local proto kernel metric 256 pref medium
multicast ff00::/8 dev vxlan.calico table local proto kernel metric 256 pref medium
multicast ff00::/8 dev eth1 table local proto kernel metric 256 pref medium

[manuelbuil]

Calico by default picks the interface randomly. If you added a new interface and restarted the node, it could have chosen the new interface. When you have multiple NICs, you should use one of the Calico methods to set the interface: https://docs.tigera.io/calico/latest/reference/installation/api#operator.tigera.io/v1.NodeAddressAutodetection

[kgts23]

Thanks @manuelbuil, this was a huge help for me.

[manuelbuil]

you are welcome!