Bind LoadBalancer nodePort for rootless k3s

[hinshun]

Is your feature request related to a problem? Please describe.
When creating a service, typically the service port is for pod-to-pod access whereas nodePort is used for external access. In non-rootless mode, I was able to access a service of type LoadBalancer via localhost on both its service port and nodePort. I assume that's unique to k3s with its Klipper embedded service load balancer?

However, in rootless mode, I was only able to access it via localhost on its service port but not its nodePort. It is described in the advanced documentation but can be confusing for new users.

https://docs.k3s.io/advanced#known-issues-with-rootless-mode

For example, a Service on port 80 will become 10080 on the host, but 8080 will become 8080 without any offset. Currently, only LoadBalancer Services are automatically bound.

Describe the solution you'd like

Ideally, in rootless-mode, service's nodePort are also binded on the host to match the UX of rootful mode.

[brandond]

I don't believe this is possible. When running k3s rootless, k3s runs within a user network namespace. Only LoadBalancer service ports are forwarded from the real host network namespace into the user's network namespace, that the rootless kubelet thinks is the host network namespace. I don't think it is desirable (or even possible) to map all the host ports.

I would defer to @AkihiroSuda as the rootlesskit maintainer; if he thinks this is possible we can reopen but at the moment I am going to close this out as a documented limitation of rootless operation.

[AkihiroSuda]

I don't think it is desirable (or even possible) to map all the host ports.

It is still possible to monitor and map all the Kubernetes node ports.

Probably just this function has to be modified to watch the node ports

k3s/pkg/rootlessports/controller.go

Lines 121 to 158 in 3e13e36

	func (h *handler) toBindPorts() (map[int]int, error) {
	svcs, err := h.serviceCache.List("", labels.Everything())
	if err != nil {
	return nil, err
	}
	toBindPorts := map[int]int{
	h.httpsPort: h.httpsPort,
	}
	if !h.enabled {
	return toBindPorts, nil
	}
	for _, svc := range svcs {
	for _, ingress := range svc.Status.LoadBalancer.Ingress {
	if ingress.IP == "" {
	continue
	}
	for _, port := range svc.Spec.Ports {
	if port.Protocol != v1.ProtocolTCP {
	continue
	}
	if port.Port != 0 {
	if port.Port <= 1024 {
	toBindPorts[10000+int(port.Port)] = int(port.Port)
	} else {
	toBindPorts[int(port.Port)] = int(port.Port)
	}
	}
	}
	}
	}
	return toBindPorts, nil
	}

[brandond]

@hinshun if you wanted to open a PR to make these changes as suggested by @AkihiroSuda, we'd be glad to review it!

[mdrahman-suse]

Validated in RC v1.29.3-rc1+k3s1

Environment

Ubuntu 22.04 Single node

lb.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-loadbalancer-pod
spec:
  selector:
    matchLabels:
      app: nginx-app-lb
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx-app-lb
    spec:
      containers:
      - name: nginx
        image: sangeetha/mytestcontainer
        ports:
        - containerPort: 80

Testing

• Follow the steps mentioned here for rootless setup: While running rootless node password file path is incorrect #3636 (comment)
• Ensure cluster is up
• Deploy loadbalancer lb.yaml
• Expose a port on the loadbalancer service less than 1024

kubectl expose deployment nginx-loadbalancer-pod --target-port=80 --type=LoadBalancer --name=myservice1 --port=81

• Get the service port and nodePort

$ kubectl get svc -o yaml
...
- apiVersion: v1
  kind: Service
  metadata:
    ...
    name: myservice1
    namespace: default
    ...
  spec:
    allocateLoadBalancerNodePorts: true
    ...
    ports:
    - nodePort: <nodePort>
      port: 81
    type: LoadBalancer

• Ensure the pod is NOT accessible from outside the cluster via port
• Ensure the pod is accessible from outside the cluster via:
  • 10000+port
  • nodePort
• Expose a port on the lb service greater than 1024

kubectl expose deployment nginx-loadbalancer-pod --target-port=80 --type=LoadBalancer --name=myservice2 --port=1182

• Get the service port and nodePort

$ kubectl get svc -o yaml
...
- apiVersion: v1
  kind: Service
  metadata:
    ...
    name: myservice2
    namespace: default
    ...
  spec:
    allocateLoadBalancerNodePorts: true
    ...
    ports:
    - nodePort: <nodePort>
      port: 1182
    type: LoadBalancer

• Ensure the pod is NOT accessible from outside the cluster via 10000+port
• Ensure the pod is accessible from outside the cluster via:
  • port
  • nodePort

Replication

$ k3s -v
k3s version v1.29.2+k3s1 (86f10213)
go version go1.21.7

port is less than 1024

• Pod is NOT accessible via port

$ curl -sL --insecure http://<external-ip>:81/name.html

• Pod is accessible via 10000+port

$ curl -sL --insecure http://<external-ip>:10081/name.html
nginx-loadbalancer-pod-7c4cdb6969-tljgf

• Pod is NOT accessible via nodePort

$ curl -sL --insecure http://<external-ip>:<nodePort>/name.html

port is greater than 1024

• Pod is NOT accessible via 10000+port

$ curl -sL --insecure http://<external-ip>:11182/name.html

• Pod is accessible via port

$ curl -sL --insecure http://<external-ip>:1182/name.html
nginx-loadbalancer-pod-7c4cdb6969-v7gv6

• Pod NOT accessible via nodePort

$ curl -sL --insecure http://<external-ip>:<nodePort>/name.html

Validation

$ k3s -v
k3s version v1.29.3-rc1+k3s1 (8aecc26b)
go version go1.21.8

port is less than 1024

• Pod is NOT accessible via port

$ curl -sL --insecure http://<external-ip>:81/name.html

• Pod accessible via 10000+port

$ curl -sL --insecure http://<external-ip>:10081/name.html
nginx-loadbalancer-pod-7c4cdb6969-rjm4b

• Pod is accessible via nodePort

$ curl -sL --insecure http://<external-ip>:<nodePort>/name.html
nginx-loadbalancer-pod-7c4cdb6969-rjm4b

port is greater than 1024

• Pod is NOT accessible via 10000+port

$ curl -sL --insecure http://<external-ip>:11182/name.html

• Pod accessible via port

$ curl -sL --insecure http://<external-ip>:1182/name.html
nginx-loadbalancer-pod-7c4cdb6969-2w8k7

• Pod is accessible via nodePort

$ curl -sL --insecure http://<external-ip>:<nodePort>/name.html
nginx-loadbalancer-pod-7c4cdb6969-2w8k7