Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

azure: implicit nic creation + public ip support #2056

Merged

Conversation

mkulke
Copy link
Contributor

@mkulke mkulke commented Sep 25, 2024

The is a rework of the networking code. Currently we manage NICs explicitly, which requires a lot of code that is prone to race conditions and resource leaks. This PR changes the code to create NICs implicitly when creating a VM, the network resource will be managed as part of the VM lifecycle.

Public IPs

There's also an opt-in toggle to attach (a implicitly-managed public ip) to the podvm. This option is mostly relevant for development purposes, there are obvious risks to expose a podvm's services to the internet in production environments. A given kubernetes deployment might not allow nodes (and hence the kata runtime) to connect to a podvm on the internet, further network configuration is required in those cases.

PodVM internet access

In the current Pod VM deployment method, the VMs rely on a feature called Default Outbound Access. This will equip a machine with a transparent public ip if no other outbound connectivity means are configured. This feature is being retired due to security considerations. Today implicit NIC-management does not support this feature any more.

A PodVM might rely on internet connectivity e.g. for guest-pull from public registries like DockerHub (note: the pod traffic itself is namespaced and routed through the node, so a Pod itself will share the outbound connectivity of the other pods in a cluster)

Outbound access should be explicitly configured, the following would be an example for AKS, where create a dedicated peerpod subnet on the Worker Node's VNet, to which we attach a NAT gateway with a public ip:

export AKS_RG="$(az aks show -n my-cluster -g my-resorce-group --query nodeResourceGroup -o tsv)"
export AZURE_VNET_NAME="$(az network vnet list -g ${AKS_RG} --query '[].name' -o tsv)"
export AKS_CIDR="$(az network vnet show -n $AZURE_VNET_NAME -g $AKS_RG --query "subnets[?name == 'aks-subnet'].addressPrefix" -o tsv)"
# 10.224.0.0/16
export MASK="${AKS_CIDR#*/}"
# 16
PEERPOD_CIDR="$(sipcalc $AKS_CIDR -n 2 | grep ^Network | grep -v current | cut -d' ' -f2)/${MASK}"
# 10.225.0.0/16
az network public-ip create -g "$AKS_RG" -n peerpod
az network nat gateway create -g "$AKS_RG" -l "$AZURE_REGION" --public-ip-addresses peerpod -n peerpod
az network vnet subnet create -g "$AKS_RG" --vnet-name "$AZURE_VNET_NAME" --nat-gateway peerpod --address-prefixes "$PEERPOD_CIDR" -n peerpod
# SUBNET_ID is to be used in the peerpod configmap
export SUBNET_ID="$(az network vnet subnet show -g "$AKS_RG" --vnet-name "$AZURE_VNET_NAME" -n peerpod --query id -o tsv)"

Successful e2e test run here

@mkulke mkulke force-pushed the mkulke/az-use-implicit-nic-creation branch from 58b9425 to 0f18ddc Compare September 25, 2024 10:12
The explicit lifecycle management of NICs is prone to errors and race
conditions, so we should rely on implicit generation and deletion of
nics (like we do with disks already).

Signed-off-by: Magnus Kulke <[email protected]>
If turned on, a public ip will be created alongside the pod VM, the
public will be provided as first IP in the `Instance.IPs` struct and
hence used by the shim to address the podvm.

Signed-off-by: Magnus Kulke <[email protected]>
This is required to allow guest-pull of dockerhub images from the podvm
as it's used in the e2e test suite.

Signed-off-by: Magnus Kulke <[email protected]>
@mkulke mkulke force-pushed the mkulke/az-use-implicit-nic-creation branch from 0f18ddc to 85d8639 Compare September 25, 2024 16:31
@mkulke mkulke marked this pull request as ready for review September 25, 2024 16:35
@mkulke mkulke requested a review from a team as a code owner September 25, 2024 16:35
Copy link
Member

@bpradipt bpradipt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@mkulke mkulke merged commit 8738939 into confidential-containers:main Sep 26, 2024
19 checks passed
@mkulke mkulke deleted the mkulke/az-use-implicit-nic-creation branch September 26, 2024 13:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants