Encrypting pod network traffic between worker node and pod VM

[bpradipt]

The default networking solution extends pod network by creating a VXLAN tunnel between worker node and pod VM.
Does this need to be encrypted? If yes what are the available mechanisms ?

[bpradipt]

Few additional questions:

1. When pod-to-pod traffic is secured via mTLS using service mesh, does it makes sense to encrypt the tunnel traffic ?
2. If the workload is TLS enabled then does it makes sense to encrypt the tunnel traffic ?

As for the options to encrypt the tunnel traffic, I could find ipsec and macsec

The fundamental question is do we really need to encrypt the tunnel traffic ?

[yoheiueda]

I think it depends on how Kubernetes pod network is configured.

In a typical case, inter-node communication of Kubernetes pod network is not encrypted. In this case, we don't need to encrypt the tunnel traffic.

If inter-node communication of Kubernetes pod network is encrypted, the tunnel traffic should be encrypted. For example, Calico supports encryption of in-cluster pod traffic. https://docs.tigera.io/calico/3.25/network-policy/encrypt-cluster-pod-traffic

[yoheiueda]

WireGuard is gaining popularity for secure tunnels.

It looks like we can implement WireGuard tunnels for CAA in a relatively straightforward way.

https://www.wireguard.com/quickstart/
https://www.wireguard.com/netns/

[bpradipt]

I think it depends on how Kubernetes pod network is configured.

In a typical case, inter-node communication of Kubernetes pod network is not encrypted. In this case, we don't need to encrypt the tunnel traffic.

If inter-node communication of Kubernetes pod network is encrypted, the tunnel traffic should be encrypted. For example, Calico supports encryption of in-cluster pod traffic. https://docs.tigera.io/calico/3.25/network-policy/encrypt-cluster-pod-traffic

@yoheiueda this makes sense. We can provide it as an optional capability for cases where pod-to-pod traffic is not encrypted by default. For cases where pod-to-pod traffic is encrypted we can use wireguard.

I'm assuming mTLS using service mesh should work as-is. Although I have personally not tested service-mesh on a peer-pods environment.

[huoqifeng]

I ever tried istio service mesh for PeerPod, yes, it works as-is because mTLS configuration all happen within service mesh proxy.

[bpradipt]

I ever tried istio service mesh for PeerPod, yes, it works as-is because mTLS configuration all happen within service mesh proxy.

Thanks for confirming @huoqifeng

[kartikjoshi21]

cc @mkulke

[mkulke]

Are we confident a tunnel over tunnel (wg over vxlan) configuration will cause no issues in some setups?

[yoheiueda]

@mkulke

WireGuard is a protocol that encapsulates IP packet as described here.

https://www.wireguard.com/#conceptual-overview

WireGuard securely encapsulates IP packets over UDP. You add a WireGuard interface, configure it with your private key and your peers' public keys, and then you send packets across it. All issues of key distribution and pushed configurations are out of scope of WireGuard; these are issues much better left for other layers, lest we end up with the bloat of IKE or OpenVPN

So, I don't think we need to encapsulate VXLAN over WireGuard for our pod network tunneling. We can just use WireGuard as a point-to-point network tunnel.

@kartikjoshi21 have you started implementing this enhancement?

BTW, I am working on refactoring our pod network code by simplifying the interface of the bitops package.

[kartikjoshi21]

@yoheiueda I started trying out wireguard, But got stuck due to some issues in connectivity from worker node -> pod vm. Yet to look into what was missing in the setup.

[yoheiueda]

@kartikjoshi21 Thank you for the status info. I think I can help you if you can share the code or experimental setup information.

As I said before, I am refactoring the network code. (#947 and #954) This is probably conflicting with your code, and I can also help you to rebase your code if my PRs get merged before yours.

[yoheiueda]

I recently investigated the feasibility of WireGuard tunnel. I did some experiments using the ip command to set up WireGuard-based tunneling for peer pods, and it worked fine.

I also noticed that WireGuard has additional benefits for networks with NAT. If a worker node is a WireGuard client, and a peer pod VM is a WireGuard server, the worker node can be behind a network with NAT.
https://www.wireguard.com/quickstart/#nat-and-firewall-traversal-persistence

This will work fine with the public IP support by this PR. #2056

The current VXLAN tunnel does not support NAT traversal, since Linux connection tracking cannot correcty handle VXLAN packet streams. #2015

[yoheiueda]

My current design of WireGuard support

[katexochen]

Why do we want to encrypt that traffic again? In the coco attacker model, the worker node is untrusted. So we are building a very secure channel to a party we don't trust anyway, protecting against what?

[yoheiueda]

@katexochen Yes. Worker nodes are not trusted, so we should not rely only on network encryption between a worker node and a peer pod VM. Applications runs on a pod need to use other protection mechanisms such as TLS for protecting data-plane traffic against worker nodes.

This optional feature is for an additional protection as defense in depth. The data-plane tunnel encryption can protect attacks from outside a Kubernetes cluster. In some use cases such as #2056 , tunnel traffic will go through the Internet. This additional protection may mitigate risks of data leakage when basic data protection is not properly configured.

Another benetit of WireGuard is the support of NAT traversal as mentioned in #688 (comment). VXLAN does not work well with NAT, and WireGuard will also be an option to support such network topologies.

@bpradipt @davidhadas Any comments?

[bpradipt]

@katexochen this is a vxlan tunnel alternative. One of the use cases that we have for cloud-api-adaptor is bursting to cloud from a development environment (eg kind cluster on a dev laptop) to run large VM or GPU VM to experiment with AI models.
Another use case that we have is enterprise focused where it is used for bursting from on-prem to cloud. Although in this case there is site-to-site vpn but we have found issues with VXLAN tunneling.

[katexochen]

So we expected better tunnel stability from switching to wg and won't nest it?