[Release-1.28] - K3s fails to start after running k3s certificate rotate-ca

[brandond]

Backport fix for K3s fails to start after running k3s certificate rotate-ca

• K3s fails to start after running k3s certificate rotate-ca #11014

[endawkins]

Validated on release-1.28 using commit 3cc4334 | version v1.28

Environment Details:

Node(s) CPU architecture, OS, and Version:

Linux ip-172-31-11-132 5.14.21-150500.55.44-default #1 SMP PREEMPT_DYNAMIC Mon Jan 15 10:03:40 UTC 2024 (cc7d8b6) x86_64 x86_64 x86_64 GNU/Linux
NAME="SLES"
VERSION="15-SP5"
VERSION_ID="15.5"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp5"
DOCUMENTATION_URL="https://documentation.suse.com/"

Cluster Configuration:

1 server (configuration does not matter)

Files:

• config.yaml

cluster-init: true
write-kubeconfig-mode: 644

• rotate-default-ca-certs.sh
  https:/k3s-io/k3s/blob/release-1.28/contrib/util/rotate-default-ca-certs.sh

Steps:

1. Install K3s
2. Update Certificates using script
3. Rotate ca-certs k3s certificate rotate-ca
4. Restart k3s sudo systemctl restart k3s
5. Check status of k3s sudo systemctl status k3s

Reproduction of the Issue:
#11014 (comment)

Validation of the Issue:

- Observations:

$ k3s -v
k3s version v1.28.14+k3s-3cc4334f (3cc4334f)
go version go1.22.6

$ ./rotate-default-ca-certs.sh
To update certificates, you may now run:
    k3s certificate rotate-ca --path=/var/lib/rancher/k3s/server/rotate-ca

$ k3s certificate rotate-ca --path=/var/lib/rancher/k3s/server/rotate-ca
certificates saved to datastore

$ sudo systemctl restart k3s
$ sudo systemctl status k3s
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
     Active: active (running) since Mon 2024-10-14 20:33:16 UTC; 14min ago
       Docs: https://k3s.io
    Process: 20267 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service 2>/dev/null (code=exited, status=0/SUCCESS)
    Process: 20269 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 20270 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 20271 (k3s-server)

$ kubectl get nodes,pods -A -o wide
NAME                                               STATUS   ROLES                       AGE    VERSION                 INTERNAL-IP     EXTERNAL-IP     OS-IMAGE                              KERNEL-VERSION                 CONTAINER-RUNTIME
node/ip-172-31-11-132.us-east-2.compute.internal   Ready    control-plane,etcd,master   139m   v1.28.14+k3s-3cc4334f   172.31.11.132   [REDACTED]      SUSE Linux Enterprise Server 15 SP5   5.14.21-150500.55.44-default   containerd://1.7.22-k3s1.28

NAMESPACE     NAME                                          READY   STATUS      RESTARTS   AGE    IP          NODE                                          NOMINATED NODE   READINESS GATES
kube-system   pod/coredns-559656f558-psdrp                  1/1     Running     0          138m   10.42.0.6   ip-172-31-11-132.us-east-2.compute.internal   <none>           <none>
kube-system   pod/helm-install-traefik-crd-p7gs6            0/1     Completed   0          138m   <none>      ip-172-31-11-132.us-east-2.compute.internal   <none>           <none>
kube-system   pod/helm-install-traefik-ztt2v                0/1     Completed   1          138m   <none>      ip-172-31-11-132.us-east-2.compute.internal   <none>           <none>
kube-system   pod/local-path-provisioner-7677785564-qbt67   1/1     Running     0          138m   10.42.0.5   ip-172-31-11-132.us-east-2.compute.internal   <none>           <none>
kube-system   pod/metrics-server-7cbbc464f4-qq8p5           1/1     Running     0          138m   10.42.0.4   ip-172-31-11-132.us-east-2.compute.internal   <none>           <none>
kube-system   pod/svclb-traefik-9584e60d-w7vqs              2/2     Running     0          138m   10.42.0.7   ip-172-31-11-132.us-east-2.compute.internal   <none>           <none>
kube-system   pod/traefik-6c7b69cd74-4474l                  1/1     Running     0          138m   10.42.0.8   ip-172-31-11-132.us-east-2.compute.internal   <none>           <none>