Memory controller properties with cgroup v2

[tchoumbe-91]

Hello,

I'm working on cgroup v2 nodes in my cluster (RHEL 8 nodes).

According to this post:
https://kubernetes.io/blog/2021/11/26/qos-memory-resources/

I should have a correspondance such as:

• container/pod memory request and memory.min in cgroup
• container/pod memory limit and memory.high in cgroup

But this is not the case, for a guaranteed pod like "rke2-coredns" :

• memory.min is equal to 0,
• memory.high is equal to "max"
• memory.max is equal to "134217728"

Am I missing something about cgroup v2 support with rke2 (v1.22) ?

[brandond]

Have you set any resource requests or limits on your pod? The cgroups will not have these settings if the pods do not.

[tchoumbe-91]

Hi @brandond thanks for your answer.

The rke2-coredns pod has the following configuration :

Containers:
  coredns:
    Container ID:  containerd://a3ceb1df759b315d66443869431181ebc5185f87a22a18f05d637811f9ed87c3
    Image:         rancher/hardened-coredns:v1.8.5-build20211119
    Image ID:      docker.io/rancher/hardened-coredns@sha256:92def4a89ef6c4a2351c82d661f306994656fb374cf4367a1d8d51caa19df2b4
    Ports:         53/UDP, 53/TCP
    Host Ports:    0/UDP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Running
      Started:      Sun, 03 Apr 2022 21:07:13 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  128Mi
    Requests:
      cpu:        100m
      memory:     128Mi

I've also tried to create a Guaranteed pod with the following configuration :

apiVersion: v1
kind: Pod
metadata:
  name: guaranteed-demo
spec:
  containers:
  - name: qos-demo-ctr
    image: index.docker.io/docker.io/library/nginx:1.21.6
    imagePullPolicy: IfNotPresent
    resources:
      limits:
        memory: "200Mi"
        cpu: "100m"
      requests:
        memory: "200Mi"
        cpu: "100m"

Yet, I don't get the expected values in the corresponding cgroup files.

Same thing with Burstable or Besteffort pod.

[brandond]

Hmm, the page you linked mentions a few requirements:
https://kubernetes.io/blog/2021/11/26/qos-memory-resources/#how-do-i-use-it

• Memory QoS in Alpha phase is designed to support containerd and cri-o. Related PR Feature: containerd-cri support LinuxContainerResources.Unified #5627 has been merged and will be released in containerd 1.6
• With those prerequisites met, you can enable the memory QoS feature gate (see Set kubelet parameters via a config file).

Are you using containerd 1.6? Have you enabled the alpha feature-gate?

[tchoumbe-91]

Sorry I missed the mismatch between containerd versions. This can explain the behaviour I am observing with the cgroup value.

How can I use my own containerd instead of the one provided by rke2 ?

[brandond]

1. Install and start containerd on your node
2. Start rke2 with --container-runtime-endpoint=/path/to/containerd/socket

[tchoumbe-91]

Thanks, I will try to use this to use Memory QoS feature.

Do you have any idea how much time there is between a RKE2 release based upon Kubernetes 1.24 and the Kubernetes 1.24 release ?

[brandond]

It depends a lot on how much ends up being changed in 1.24, and whether it breaks anything in RKE2 or its packaged components. We generally aim for a release within ~2 weeks of the upstream release, but it may be as long as a month if we are waiting for other projects that we bundle in to become compatible.

[tchoumbe-91]

Thanks for these informations.

To avoid troubles that I observed on RHEL with systemd version, I tried on a Fedora node for testing.

Then, I managed to run a rke2-agent node with the following elements :

• containerd 1.6 installation as a service
• RKE2 container-runtime-endpoint set
• MemoryQoS feature-gate activation

Here is the corresponding logs form journald :

[root@cwp01sv ~]# journalctl -u rke2-agent -f
Apr 06 11:11:41 cwp01sv systemd[1]: Starting Rancher Kubernetes Engine v2 (agent)...
Apr 06 11:11:41 cwp01sv sh[4181]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Apr 06 11:11:41 cwp01sv sh[4182]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Apr 06 11:11:41 cwp01sv systemd[1]: rke2-agent.service: Found left-over process 1180 (containerd-shim) in control group while starting unit. Ignoring.
Apr 06 11:11:41 cwp01sv systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Apr 06 11:11:41 cwp01sv systemd[1]: rke2-agent.service: Found left-over process 1182 (containerd-shim) in control group while starting unit. Ignoring.
Apr 06 11:11:41 cwp01sv systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Apr 06 11:11:41 cwp01sv systemd[1]: rke2-agent.service: Found left-over process 1184 (containerd-shim) in control group while starting unit. Ignoring.
Apr 06 11:11:41 cwp01sv systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Apr 06 11:11:41 cwp01sv systemd[1]: rke2-agent.service: Found left-over process 2163 (containerd-shim) in control group while starting unit. Ignoring.
Apr 06 11:11:41 cwp01sv systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Apr 06 11:11:41 cwp01sv rke2[4185]: time="2022-04-06T11:11:41-04:00" level=warning msg="not running in CIS mode"
Apr 06 11:11:41 cwp01sv rke2[4185]: time="2022-04-06T11:11:41-04:00" level=info msg="Starting rke2 agent v1.22.7+rke2r1 (2bc7ff9bbd4109f7df3ae25a22b241d27b0370a9)"
Apr 06 11:11:41 cwp01sv rke2[4185]: time="2022-04-06T11:11:41-04:00" level=info msg="Running load balancer 127.0.0.1:6444 -> [200.64.43.3:9345 srv01sv:9345]"
Apr 06 11:11:41 cwp01sv rke2[4185]: time="2022-04-06T11:11:41-04:00" level=warning msg="Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash. Use the full token from the server's node-token file to enable Cluster CA validation."
Apr 06 11:11:41 cwp01sv rke2[4185]: time="2022-04-06T11:11:41-04:00" level=info msg="Running load balancer 127.0.0.1:6443 -> [200.64.43.3:6443 srv01sv:6443]"
Apr 06 11:11:42 cwp01sv rke2[4185]: time="2022-04-06T11:11:42-04:00" level=info msg="Module overlay was already loaded"
Apr 06 11:11:42 cwp01sv rke2[4185]: time="2022-04-06T11:11:42-04:00" level=info msg="Module nf_conntrack was already loaded"
Apr 06 11:11:42 cwp01sv rke2[4185]: time="2022-04-06T11:11:42-04:00" level=info msg="Module br_netfilter was already loaded"
Apr 06 11:11:42 cwp01sv rke2[4185]: time="2022-04-06T11:11:42-04:00" level=info msg="Module iptable_nat was already loaded"
Apr 06 11:11:42 cwp01sv rke2[4185]: time="2022-04-06T11:11:42-04:00" level=info msg="Runtime image index.docker.io/rancher/rke2-runtime:v1.22.7-rke2r1 bin and charts directories already exist; skipping extract"
Apr 06 11:11:42 cwp01sv rke2[4185]: time="2022-04-06T11:11:42-04:00" level=info msg="Connecting to proxy" url="wss://200.64.43.3:9345/v1-rke2/connect"
Apr 06 11:11:42 cwp01sv rke2[4185]: time="2022-04-06T11:11:42-04:00" level=info msg="Running kubelet --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/containerd/containerd.sock --container-runtime=remote --containerd=/run/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --feature-gates=MemoryQoS=true --healthz-bind-address=127.0.0.1 --hostname-override=cwp01sv --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file-max-size=50 --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --logtostderr=false --network-plugin=cni --node-ip=200.64.43.1 --node-labels= --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --read-only-port=0 --resolv-conf=/tmp/rke2-resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key"
Apr 06 11:11:42 cwp01sv rke2[4185]: time="2022-04-06T11:11:42-04:00" level=info msg="Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=cwp01sv --kubeconfig=/var/lib/rancher/rke2/agent/kubeproxy.kubeconfig --proxy-mode=iptables"
Apr 06 11:11:42 cwp01sv rke2[4185]: time="2022-04-06T11:11:42-04:00" level=info msg="labels have been set successfully on node: cwp01sv"
Apr 06 11:11:42 cwp01sv systemd[1]: Started Rancher Kubernetes Engine v2 (agent).

I load a Guaranteed pod described as follow :

apiVersion: v1
kind: Pod
metadata:
  name: guaranteed-demo
spec:
  containers:
  - name: qos-demo-ctr
    image: index.docker.io/docker.io/library/nginx:1.21.6
    imagePullPolicy: IfNotPresent
    resources:
      limits:
        memory: "20Mi"
        cpu: "100m"
      requests:
        memory: "20Mi"
        cpu: "100m"

The pod is running well :

[root@srv01sv ~]# /var/lib/rancher/rke2/bin/kubectl describe pod guaranteed-demo -n qos-example
Name:         guaranteed-demo
Namespace:    qos-example
Priority:     0
Node:         cwp01sv/200.64.43.1
Start Time:   Mon, 11 Apr 2022 10:37:52 +0000
Labels:       <none>
Annotations:  <none>
Status:       Running
IP:           10.42.1.7
IPs:
  IP:  10.42.1.7
Containers:
  qos-demo-ctr:
    Container ID:   containerd://1439d9f5a320177939f2e909cbee95cc747be9a521b008c6ded649ad51f8ca73
    Image:          index.docker.io/docker.io/library/nginx:1.21.6
    Image ID:       docker.io/docker.io/library/nginx@sha256:83d487b625d8c7818044c04f1b48aabccd3f51c3341fc300926846bca0c439e6
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Mon, 11 Apr 2022 10:37:54 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  20Mi
    Requests:
      cpu:        100m
      memory:     20Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-smghn (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kube-api-access-smghn:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Guaranteed
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

Exploring the cgroup files :

[root@cwp01sv ~]# POD_ID="$(/var/lib/rancher/rke2/bin/crictl --runtime-endpoint unix:///run/containerd/containerd.sock pods --name guaranteed-demo -q)"
[root@cwp01sv ~]# /var/lib/rancher/rke2/bin/crictl --runtime-endpoint unix:///run/containerd/containerd.sock inspectp -o=json $POD_ID | grep cgroupsPath
        "cgroupsPath": "kubepods-podc63ecbaa_76aa_4832_98b0_12d24a8c053f.slice:cri-containerd:1dd4859e5b30f6089f7f81f807828091ec83f1f6693d7215ea7c8c4be4eb486d",
[root@cwp01sv ~]# [root@cwp01sv ~]# cd /sys/fs/cgroup/kubepods.slice/kubepods-podc63ecbaa_76aa_4832_98b0_12d24a8c053f.slice/cri-containerd-1439d9f5a320177939f2e909cbee95cc747be9a521b008c6ded649ad51f8ca73.scope/
[root@cwp01sv cri-containerd-1439d9f5a320177939f2e909cbee95cc747be9a521b008c6ded649ad51f8ca73.scope]# cat memory.min
20971520
[root@cwp01sv cri-containerd-1439d9f5a320177939f2e909cbee95cc747be9a521b008c6ded649ad51f8ca73.scope]# cat memory.low
0
[root@cwp01sv cri-containerd-1439d9f5a320177939f2e909cbee95cc747be9a521b008c6ded649ad51f8ca73.scope]# cat memory.high
max
[root@cwp01sv cri-containerd-1439d9f5a320177939f2e909cbee95cc747be9a521b008c6ded649ad51f8ca73.scope]# cat memory.max
20971520

Then, we have :

• memory.min = container.request.memory, consistent with the Memory QoS feature post
• memory.high = 0 consistent with the Memory QoS feature post:
  (defaults value as throttling factor * container.memory.limit > container.memory.request),
• memory.max = container.limit.memory, consistent

For a burstable pod as follows :

apiVersion: v1
kind: Pod
metadata:
  name: burstable-demo
spec:
  containers:
  - name: burstable-nginx
    image: index.docker.io/docker.io/library/nginx:1.21.6
    imagePullPolicy: IfNotPresent
    resources:
      limits:
        memory: "200Mi"
      requests:
        memory: "100Mi"

Then, we have :

• memory.min = container.request.memory, consistent with the Memory QoS feature post
• memory.high = 167772160, consistent with the Memory QoS feature post: default throttling factor = 0.8.
• memory.max = container.limit.memory, consistent

So, it seems like working to me.

Thanks for the support !