Air-Gap environment failed to pull images from insecure private registry

[bluefish1976]

Environmental Info:
RKE2 Version:
rke2 version v1.28.11+rke2r1 (6b12d7a)
go version go1.21.11 X:boringcrypto

Node(s) CPU architecture, OS, and Version:
Linux kubecp01 5.14.0-362.8.1.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Oct 3 11:12:36 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux
Red Hat 9.3

Cluster Configuration:
I have a test environment with 1 server and 2 nodes

Describe the bug:
I can't pull images from insecure private registry using rke2, with podman I can pull images from the private registry.
I have created the file /etc/rancher/rke2/registries.yaml with the configuration of my private registry

Steps To Reproduce:

• Installed RKE2: I have installed rke2 in an Air-gap environment as described in the official web page: https://docs.rke2.io/install/airgap. Everything works as expected but when I'm installing rancher: helm install rancher ./rancher-.tgz .... (https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/install-rancher-ha#2-install-rancher) images can't be pulled and if I describe the cert-manager pods I see the following error: Warning Failed 6h13m (x4 over 6h15m) kubelet Failed to pull image "registry.registry.svc.cluster.local:5000/quay.io/jetstack/cert-manager-controller:v1.15.3": failed to pull and unpack image "registry.registry.svc.cluster.local:5000/quay.io/jetstack/cert-manager-controller:v1.15.3": failed to resolve reference "registry.registry.svc.cluster.local:5000/quay.io/jetstack/cert-manager-controller:v1.15.3": failed to do request: Head "https://registry.registry.svc.cluster.local:5000/v2/quay.io/jetstack/cert-manager-controller/manifests/v1.15.3": http: server gave HTTP response to HTTPS client

Expected behavior:
I will expect that pods would be deployed

Actual behavior:
Pods are not deployed and are always in the status: ContainerCreating

Additional context / logs:
The file /etc/rancher/rke2/registries.yaml has the following configuration:

mirrors:
  "registry.registry.svc.cluster.local:5000":
    endpoint:
      - "http://registry.registry.svc.cluster.local:5000"
configs:
  "registry.registry.svc.cluster.local:5000":
    tls:
      insecure_skip_verify: true

I don't understand why if the registry it's configured with http rke2 keeps trying https

[brandond]

Note that specifying TLS options for a registry that is not using TLS doesn't make much sense. Why have you configured it to skip TLS certificate verification if your registry isn't using TLS? That should only be used to allow an untrusted certificate, not to disable TLS.

Containerd always falls back to trying the default endpoint, which is always https. This means that it will try pulling from http://registry.registry.svc.cluster.local:5000 first, and if the pull fails, it will fall back to trying https://registry.registry.svc.cluster.local:5000

Check containerd.log to see why it's unable to pull the image from your registry over http. The kubelet only ever shows the terminal pull failure, not the failure to pull from the mirror endpoints.

[bluefish1976]

Hello @brandond thank you for your quick response. I thought that the TLS configuration was so that it did not search for https, not that it was to ignore an insecure certificate. I'm doing the lab again with the same steps and will remove the tls configuration. I will reopen again if I have the same issue.
Thank you very much for your time.