Add multi IP support for SBR #623

anuragensemble · 2021-04-27T07:07:50Z

[sbr]: Use different tableID for every ipCfg
Move default table routes which match the ipCfg config

This allows SBR plugin to accommodate for multi-ip interfaces

Fixes #581

Signed-off-by: Anurag Dwivedi [email protected]

Move default table routes which match the ipCfg config This allows SBR plugin to accommodate for multi-ip interfaces Fixes containernetworking#581 Signed-off-by: Anurag Dwivedi <[email protected]>

squeed · 2021-04-28T15:36:27Z

hey @plwhite can you give this a look-over? Thanks!

plwhite · 2021-05-03T18:15:45Z

Sorry, I've been out of office for a couple of days, but will get to this in the next day or so.

squeed · 2021-05-05T15:25:06Z

You need to fix gofmt, FYI.

plwhite · 2021-05-07T12:32:43Z

plugins/meta/sbr/sbr_linux_test.go

+ Table: 101,
+ LinkIndex: expNet1.Routes[i].LinkIndex})
+ } else {
+ // All 192.168.1.x routes expected in tabele 100


Trivial typo: "tabele" should be "table"

plwhite · 2021-05-07T12:34:41Z

plugins/meta/sbr/sbr_linux_test.go

+ Table: 101,
+ LinkIndex: expNet1.Routes[0].LinkIndex})
+
+ // 2 Rules will ve created for each IP address. (100, 101)


Trivial typo: "ve" should be "be"

plwhite · 2021-05-07T12:46:01Z

plugins/meta/sbr/main.go


+ // Use a different table for each ipCfg
+ table++


In the other places where a table ID is generated there are a few lines of code to make sure that the table isn't already in use. I think you strictly ought to do that here - i.e. take the little bit of code around firstTableID and turn it into a simple subroutine so it does the same checks in both places.

This is unlikely to go wrong, but if it does it would be horrible to have to figure out.

plwhite · 2021-05-07T13:01:03Z

Sorry for the delay - generally looks good to me. I've added a few notes, mostly about comments but one minor code issue.

I think this merits updating the docs at https://www.cni.dev/plugins/current/meta/sbr/ - just a couple of lines explaining what the expected behaviour for routes and rules is with multiple IP addresses on an interface.

anuragensemble · 2021-05-12T06:18:31Z

Thanks @squeed and @plwhite . Was away for a few days. Will update the request in a day or two.

Check tableID not in use for every ipCfg This allows SBR plugin to accommodate for multi-ip interfaces Fixes containernetworking#581 Signed-off-by: Anurag Dwivedi <[email protected]>

anuragensemble · 2021-05-15T14:16:47Z

Hi, @squeed and @plwhite , I have updated the review. Please review at your convenience.

Post confirmation I can add a few lines in the doc about multi-ip scenario (The behavior remains consistent with the current definition, just a footnote about separating the routes meant for different IPs on a single interface into different routing tables as well. )

plwhite · 2021-05-17T19:02:31Z

LGTM - thanks for the updates.

squeed · 2021-05-19T15:19:20Z

/lgtm (assuming CI passes)

dcbw · 2021-05-19T15:33:47Z

Thanks!

svallala · 2021-05-20T08:30:54Z

@anuragensemble @plwhite @squeed, seems like an issue with this check in. I have a Pod with 2 interfaces.

[root@c-2-server-01 /]# ip -6 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
4: eth0@if325: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP
inet6 fd74:ca9b:3a09:868c:172:18:0:41c/128 scope global
valid_lft forever preferred_lft forever
inet6 fe80::2881:62ff:fe79:14ab/64 scope link
valid_lft forever preferred_lft forever
39: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 state UP qlen 1000
inet6 fd74:ca9b:3a09:868e:c0:a8:c:71/64 scope global
valid_lft forever preferred_lft forever
inet6 fd74:ca9b:3a09:8660:c0:a8:1:70/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::84d7:58ff:fe91:384/64 scope link
valid_lft forever preferred_lft forever

Each of the IPs here has a default gateway, rightly eth0 is added to the default gateway

[root@c-2-server-01 /]# ip -6 route
fd74:ca9b:3a09:868c:172:18:0:41c dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via fe80::ecee:eeff:feee:eeee dev eth0 metric 1024 pref medium

Each IP subnet for interface eth1, has a default gateway and rightly there are two additional routing tables, that is what your change is fixing.

[root@c-2-server-01 /]# ip -6 rule
0: from all lookup local
32764: from fd74:ca9b:3a09:868e:c0:a8:c:71 lookup 101
32765: from fd74:ca9b:3a09:8660:c0:a8:1:70 lookup 100
32766: from all lookup main

[root@c-2-server-01 /]# ip -6 route show table 100
fd74:ca9b:3a09:8660::/64 dev eth1 proto kernel metric 256 pref medium
fd74:ca9b:3a09:868e::/64 dev eth1 proto kernel metric 256 pref medium
fe80::/64 dev eth1 proto kernel metric 256 pref medium
default via fd74:ca9b:3a09:8660::1 dev eth1 metric 1024 pref medium

[root@c-2-server-01 /]# ip -6 route show table 101
fd74:ca9b:3a09:8660::/64 dev eth1 proto kernel metric 256 pref medium
fd74:ca9b:3a09:868e::/64 dev eth1 proto kernel metric 256 pref medium
fe80::/64 dev eth1 proto kernel metric 256 pref medium
default via fd74:ca9b:3a09:868e::1 dev eth1 metric 1024 pref medium

The issue here is that for table 100 where the source lookup is fd74:ca9b:3a09:8660:c0:a8:1:70, it should not have a route entry for fd74:ca9b:3a09:868e::/64 subnet, same is the case with table 101.

I think traffic will still flow fine, but it is not correct that the directly connected networks are spanning both tables. I stand corrected if this is expected behavior since the interface does have the subnet.

Regards
Shravan

anuragensemble · 2021-05-23T11:51:18Z

@svallala , Can you please share the network-attachment-definition spec for the above case. In the recent update we are performing subnet matching on the routes added by the previous plugin (+ by interface creation) and moving them into their respective routing tables.

Requesting the full NAD spec to make a more informed comment about the validity of the logic.

svallala · 2021-05-25T08:39:57Z

@anuragensemble, please find the NAD. If you are looking for the IP addresses allocated, you might not get that information. Sending you the IPAM response that the plugin would have received.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
annotations:
k8s.v1.cni.cncf.io/resourceName: intel.com/enp175s0f0_iavf
creationTimestamp: "2021-05-25T04:44:37Z"
generation: 1
labels:
robin.io/cni: sriov
robin.io/domain: ROBIN
managedFields:

apiVersion: k8s.cni.cncf.io/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:k8s.v1.cni.cncf.io/resourceName: {}
f:labels:
.: {}
f:robin.io/cni: {}
f:robin.io/domain: {}
f:spec:
.: {}
f:config: {}
manager: kubectl-create
operation: Update
time: "2021-05-25T04:44:37Z"
name: c-1-server-01-net1
namespace: t001-u000003
resourceVersion: "1537546"
uid: b38b79ba-639a-4e00-b609-026f6eeef145
spec:
config: '{ "cniVersion": "0.3.1", "name": "sriov-network", "plugins": [ { "type":
"sriov", "vlan": 20, "ippool": "sriov7", "ipam": { "type": "robin-ipam" }, {"type": "sbr"} ] }'

IPAM Response

{"cniVersion": "0.3.1", "dns": {}, "ips": [{"version": "6", "address": "fd74:ca9b:3a09:8687:00c0:00a8:0001:0089/64", "gateway": "fd74:ca9b:3a09:8687:00c0:00a8:0001:0001"}, {"version": "6", "address": "fd74:ca9b:3a09:868e:00c0:00a8:000b:009c/64", "gateway": "fd74:ca9b:3a09:868e::0001"}]}

svallala · 2022-01-25T07:51:54Z

@anuragensemble I don't see the subnet check in the code before adding the routes. Can you point me to the snippet where the subnet check is performed.

I still see following behavior with the latest code

4: eth0@if15806: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP
inet6 fd74:ca9b:3a09:868c:172:18:0:7b06/128 scope global
valid_lft forever preferred_lft forever
inet6 fe80::b024:2cff:fef5:a2ed/64 scope link
valid_lft forever preferred_lft forever
80: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 state UP qlen 1000
inet6 fd74:ca9b:3a09:868d:c0:a8:b:91/64 scope global
valid_lft forever preferred_lft forever
inet6 fd74:ca9b:3a09:868a:c0:a8:1:6d/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::78cc:44ff:fec7:c19a/64 scope link
valid_lft forever preferred_lft forever

[root@sbr-1-9899bcc95-bhb9j pktgen]# ip -6 route
fd74:ca9b:3a09:868c:172:18:0:7b06 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via fe80::ecee:eeff:feee:eeee dev eth0 metric 1024 pref medium
[root@sbr-1-9899bcc95-bhb9j pktgen]# ip -6 rule
0: from all lookup local
32764: from fd74:ca9b:3a09:868d:c0:a8:b:91 lookup 101
32765: from fd74:ca9b:3a09:868a:c0:a8:1:6d lookup 100
32766: from all lookup main

[root@sbr-1-9899bcc95-bhb9j pktgen]# ip -6 route show table 100
fd74:ca9b:3a09:868a::/64 dev eth1 proto kernel metric 256 mtu 9000 pref medium
fd74:ca9b:3a09:868d::/64 dev eth1 proto kernel metric 256 mtu 9000 pref medium
fe80::/64 dev eth1 proto kernel metric 256 mtu 9000 pref medium

[root@sbr-1-9899bcc95-bhb9j pktgen]# ip -6 route show table 101
fd74:ca9b:3a09:868a::/64 dev eth1 proto kernel metric 256 mtu 9000 pref medium
fd74:ca9b:3a09:868d::/64 dev eth1 proto kernel metric 256 mtu 9000 pref medium
fe80::/64 dev eth1 proto kernel metric 256 mtu 9000 pref medium
default via fd74:ca9b:3a09:868d::1 dev eth1 metric 1024 pref medium

As you can see the last two tables have a route entry that should not be present.

[sbr]: Use different tableID for every ipCfg

f34c600

Move default table routes which match the ipCfg config This allows SBR plugin to accommodate for multi-ip interfaces Fixes containernetworking#581 Signed-off-by: Anurag Dwivedi <[email protected]>

anuragensemble marked this pull request as ready for review April 27, 2021 07:09

plwhite reviewed May 7, 2021

View reviewed changes

[sbr]: Use different tableID for every ipCfg

f72aa98

Check tableID not in use for every ipCfg This allows SBR plugin to accommodate for multi-ip interfaces Fixes containernetworking#581 Signed-off-by: Anurag Dwivedi <[email protected]>

dcbw merged commit 6618a0a into containernetworking:master May 19, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add multi IP support for SBR #623

Add multi IP support for SBR #623

anuragensemble commented Apr 27, 2021 •

edited

Loading

squeed commented Apr 28, 2021

plwhite commented May 3, 2021

squeed commented May 5, 2021

plwhite May 7, 2021

plwhite May 7, 2021

plwhite May 7, 2021

plwhite commented May 7, 2021

anuragensemble commented May 12, 2021

anuragensemble commented May 15, 2021

plwhite commented May 17, 2021

squeed commented May 19, 2021

dcbw commented May 19, 2021

svallala commented May 20, 2021 •

edited

Loading

anuragensemble commented May 23, 2021

svallala commented May 25, 2021 •

edited

Loading

svallala commented Jan 25, 2022

Add multi IP support for SBR #623

Add multi IP support for SBR #623

Conversation

anuragensemble commented Apr 27, 2021 • edited Loading

squeed commented Apr 28, 2021

plwhite commented May 3, 2021

squeed commented May 5, 2021

plwhite May 7, 2021

Choose a reason for hiding this comment

plwhite May 7, 2021

Choose a reason for hiding this comment

plwhite May 7, 2021

Choose a reason for hiding this comment

plwhite commented May 7, 2021

anuragensemble commented May 12, 2021

anuragensemble commented May 15, 2021

plwhite commented May 17, 2021

squeed commented May 19, 2021

dcbw commented May 19, 2021

svallala commented May 20, 2021 • edited Loading

anuragensemble commented May 23, 2021

svallala commented May 25, 2021 • edited Loading

IPAM Response

svallala commented Jan 25, 2022

anuragensemble commented Apr 27, 2021 •

edited

Loading

svallala commented May 20, 2021 •

edited

Loading

svallala commented May 25, 2021 •

edited

Loading