RKE2 upgrade from v1.27.10+rk2r1 to v1.28.8+rke2r1 fails

[shindebshekhar]

Environmental Info:
RKE2 Version: v1.27.10+rk2r1
Rancher Version: v2.8.3

:~ # rke2 -v
rke2 version v1.27.10+rke2r1
go version go1.20.13 X:boringcrypto

Node(s) CPU architecture, OS, and Version:

Linux tesh0m000007 5.3.18-150300.59.144-default #1 SMP Tue Dec 5 15:20:50 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

3 Master 3 Worker nodes

Describe the bug:

We are trying to upgrade from v1.27.10+rk2r1 to v1.28.8+rke2r1
During master upgrade we get below error in rke2-server journal logs
Logs:
Apr 18 10:26:55 rke2[6072]: time="2024-04-18T10:26:55+02:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Apr 18 10:26:57 rke2[6072]: time="2024-04-18T10:26:57+02:00" level=fatal msg="Failed to apply network policy default-network-ingress-webhook-policy to namespace kube-system: Internal error occurred: failed calling webhook "rancher.cattle.io.namespaces": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/namespaces?timeout=10s\": context deadline exceeded"
Apr 18 10:26:57 6 systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE

Steps To Reproduce:

1. Upload the required Images to private registry https:/rancher/rke2/releases/tag/v1.28.8%2Brke2r1
2. Download the required artifacts and copy to each of the node https:/rancher/rke2/releases/tag/v1.28.8%2Brke2r1
3. Start upgrade procedure of 1st Master node. It fails giving below error

Apr 18 10:26:55 rke2[6072]: time="2024-04-18T10:26:55+02:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Apr 18 10:26:57 rke2[6072]: time="2024-04-18T10:26:57+02:00" level=fatal msg="Failed to apply network policy default-network-ingress-webhook-policy to namespace kube-system: Internal error occurred: failed calling webhook "rancher.cattle.io.namespaces": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/namespaces?timeout=10s\": context deadline exceeded"
Apr 18 10:26:57 6 systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE

Expected behavior:

RKE2 server service should start and we should see the rke2 version getting upgraded

Actual behavior:

Startup of rke2 server service fails. Also found that kube-proxy.yaml in /var/lib/rancher/rke2/agent/pod-manifests goes missing

Additional context / logs:
Apr 18 10:26:55 rke2[6072]: time="2024-04-18T10:26:55+02:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Apr 18 10:26:57 rke2[6072]: time="2024-04-18T10:26:57+02:00" level=fatal msg="Failed to apply network policy default-network-ingress-webhook-policy to namespace kube-system: Internal error occurred: failed calling webhook "rancher.cattle.io.namespaces": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/namespaces?timeout=10s\": context deadline exceeded"
Apr 18 10:26:57 6 systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE

[brandond]

Duplicate of #5693 - see the discussion on that issue.