Kafka broker is giving us SSL handshake issue, Pods are restarting frequently

[vperi1730]

Hi @scholzj - We have a 3 pod kafka cluster that is hosted within K8's, the cluster certs are active till Apr 2025, however the kafka brokers seems to be expired. Need your help to let us know how to resolve the issue and also not sure why the actual certs and broker certs has both diff start and end dates.

Attaching the logs for more details, Please do the needful

all 3 kafka pod logs.txt

[scholzj]

What does your operator log say?

[vperi1730]

strimzi-operator-77dbf4c687-gzk48.txt

It explains about reconciliation issue

[scholzj]

Right, so that explains it, or? It is stuck and cannot operate the cluster because you didn't upgraded in time and that is why it failed to renew the certificates.

[vperi1730]

Oh ok, how do we resolve this issue now?, any recommended steps

Generally all the certs should ideally hold the same start and end date, isn't it?, why the brokers are expired before the kafka cluster certs

[scholzj]

Well, first thing will be to get the operator upgraded. Then you probably should delete all the Secrets, Pods, StatefulSets / StrimziPodSets and have Strimzi recreate them. You might also need to upgrade Kafka as part of it depending on what versions are you using.