Unable to connect to redpanda with local kubernetes cluster

[tqwewe]

I've been having issues connecting to RedPanda running in a local Kubernetes cluster both with Minikube (drivers: docker, hyperkit, virtualbox), and Docker-for-Desktop Kubernetes. I've only managed to get it working once, and that was with a Kubernetes cluster hosted on Google Cloud Platform.

The tutorial I've been following has been: https://vectorized.io/docs/quick-start-kubernetes/

And the steps I take are:

1. Install cert-manager: follow the commands shown, but with version 1.6.1 of cert manager
2. Use Helm to install Redpanda operator: follow the commands shown exactly
3. Install External connectivity: rather than install one-node-cluster, I install using the external connectivity configuration.
4. Finally once everything is setup, from my host machine (mac), I try to list topics with the external port shown from kubectl get svc: rpk --brokers 127.0.0.1:31595 topic list

I've tried the one-node-cluster many times but recently have just tried skipping straight to the external configuration, which is the one that worked on Google Cloud Platform.

Once everything is setup, when trying to list topics I get the following output:

$ rpk --brokers 127.0.0.1:31595 topic list -v
[DEBUG] opening connection to broker; addr: 127.0.0.1:31595, broker: seed 0
[DEBUG] connection opened to broker; addr: 127.0.0.1:31595, broker: seed 0
[DEBUG] issuing api versions request; broker: seed 0, version: 3
[DEBUG] wrote ApiVersions v3; broker: seed 0, bytes_written: 30, write_wait: 39.31µs, time_to_write: 22.084µs, err: <nil>
[DEBUG] read ApiVersions v3; broker: seed 0, bytes_read: 0, read_wait: 55.36µs, time_to_read: 904.057µs, err: EOF
[ERROR] unable to request api versions; broker: seed 0, err: EOF
[DEBUG] connection initialization failed; addr: 127.0.0.1:31595, broker: seed 0, err: EOF
[DEBUG] opening connection to broker; addr: 127.0.0.1:31595, broker: seed 0
[DEBUG] connection opened to broker; addr: 127.0.0.1:31595, broker: seed 0
[DEBUG] issuing api versions request; broker: seed 0, version: 3
[DEBUG] wrote ApiVersions v3; broker: seed 0, bytes_written: 30, write_wait: 27.216µs, time_to_write: 27.776µs, err: <nil>
[DEBUG] read ApiVersions v3; broker: seed 0, bytes_read: 0, read_wait: 64.188µs, time_to_read: 1.796244ms, err: EOF
[ERROR] unable to request api versions; broker: seed 0, err: EOF
[DEBUG] connection initialization failed; addr: 127.0.0.1:31595, broker: seed 0, err: EOF
[DEBUG] opening connection to broker; addr: 127.0.0.1:31595, broker: seed 0
[DEBUG] connection opened to broker; addr: 127.0.0.1:31595, broker: seed 0
[DEBUG] issuing api versions request; broker: seed 0, version: 3
[DEBUG] wrote ApiVersions v3; broker: seed 0, bytes_written: 30, write_wait: 12.845µs, time_to_write: 24.937µs, err: <nil>
[DEBUG] read ApiVersions v3; broker: seed 0, bytes_read: 0, read_wait: 45.727µs, time_to_read: 1.723373ms, err: EOF
[ERROR] unable to request api versions; broker: seed 0, err: EOF
[DEBUG] connection initialization failed; addr: 127.0.0.1:31595, broker: seed 0, err: EOF
[DEBUG] opening connection to broker; addr: 127.0.0.1:31595, broker: seed 0
[DEBUG] connection opened to broker; addr: 127.0.0.1:31595, broker: seed 0
[DEBUG] issuing api versions request; broker: seed 0, version: 3
[DEBUG] wrote ApiVersions v3; broker: seed 0, bytes_written: 30, write_wait: 20.105µs, time_to_write: 25.42µs, err: <nil>
[DEBUG] read ApiVersions v3; broker: seed 0, bytes_read: 0, read_wait: 64.334µs, time_to_read: 1.744279ms, err: EOF
[ERROR] unable to request api versions; broker: seed 0, err: EOF
[DEBUG] connection initialization failed; addr: 127.0.0.1:31595, broker: seed 0, err: EOF
[DEBUG] opening connection to broker; addr: 127.0.0.1:31595, broker: seed 0
[DEBUG] connection opened to broker; addr: 127.0.0.1:31595, broker: seed 0
[DEBUG] issuing api versions request; broker: seed 0, version: 3
[DEBUG] wrote ApiVersions v3; broker: seed 0, bytes_written: 30, write_wait: 23.404µs, time_to_write: 39.192µs, err: <nil>
[DEBUG] read ApiVersions v3; broker: seed 0, bytes_read: 0, read_wait: 52.338µs, time_to_read: 1.932385ms, err: EOF
[ERROR] unable to request api versions; broker: seed 0, err: EOF
[DEBUG] connection initialization failed; addr: 127.0.0.1:31595, broker: seed 0, err: EOF
unable to request metadata: EOF

With the main error being:

unable to request metadata: EOF

This error seems to only happen on my local setup, but does not occur when I set it up on Google Cloud Platform.

I am using the latest version of redpanda, kubernetes, docker, etc.
There's been a long discussion on Slack which may be helpful to reference: https://vectorizedcommunity.slack.com/archives/C01AJDUT88N/p1638358878411800

Some additional information:

$ kubectl get pod -o=wide

$ kubectl get pod -o=wide
NAME                      READY   STATUS    RESTARTS   AGE   IP          NODE             NOMINATED NODE   READINESS GATES
external-connectivity-0   2/2     Running   0          25m   10.1.0.15   docker-desktop   <none>           <none>
external-connectivity-1   0/2     Pending   0          25m   <none>      <none>           <none>           <none>
external-connectivity-2   0/2     Pending   0          25m   <none>      <none>           <none>           <none>

$ kubectl describe pod external-connectivity-0

$ kubectl describe pod external-connectivity-0
Name:         external-connectivity-0
Namespace:    default
Priority:     0
Node:         docker-desktop/192.168.65.4
Start Time:   Sat, 04 Dec 2021 18:40:57 +0700
Labels:       app.kubernetes.io/component=redpanda
              app.kubernetes.io/instance=external-connectivity
              app.kubernetes.io/name=redpanda
              controller-revision-hash=external-connectivity-d95c7cdbc
              statefulset.kubernetes.io/pod-name=external-connectivity-0
Annotations:  <none>
Status:       Running
IP:           10.1.0.15
IPs:
  IP:           10.1.0.15
Controlled By:  StatefulSet/external-connectivity
Init Containers:
  redpanda-configurator:
    Container ID:   docker://445024fbf1ae9c12375ac9ee72d919f1d57760997081b4ed1959b608222d1316
    Image:          vectorized/configurator:v21.10.2
    Image ID:       docker-pullable://vectorized/configurator@sha256:10041e6c86536ebab77b36b51b328aa897b34b7e0df659ec81d17684004f9455
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 04 Dec 2021 18:41:13 +0700
      Finished:     Sat, 04 Dec 2021 18:41:13 +0700
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  2Gi
    Requests:
      cpu:     1
      memory:  2Gi
    Environment:
      SERVICE_FQDN:                     external-connectivity.default.svc.cluster.local.
      CONFIG_SOURCE_DIR:                /mnt/operator
      CONFIG_DESTINATION:               /etc/redpanda/redpanda.yaml
      REDPANDA_RPC_PORT:                33145
      NODE_NAME:                         (v1:spec.nodeName)
      EXTERNAL_CONNECTIVITY:            true
      EXTERNAL_CONNECTIVITY_SUBDOMAIN:
      HOST_PORT:                        31595
      PROXY_HOST_PORT:                  31924
    Mounts:
      /etc/redpanda from config-dir (rw)
      /mnt/operator from configmap-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-m8xn5 (ro)
Containers:
  redpanda:
    Container ID:  docker://b07d71f179d5aa10de536a2318423ceaa9fd37b43e62c32a4f14dda46f606df1
    Image:         vectorized/redpanda:latest
    Image ID:      docker-pullable://vectorized/redpanda@sha256:65531e3c16b313891db67a5c6cf2839569c845840e8884966c64a6d7611d45c7
    Ports:         33145/TCP, 9644/TCP, 9092/TCP, 8082/TCP, 9093/TCP, 8083/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP, 31595/TCP, 31924/TCP
    Command:
      /usr/bin/rpk
    Args:
      redpanda
      start
      --check=false
      --advertise-rpc-addr=$(POD_NAME).external-connectivity.default.svc.cluster.local.:33145
      --overprovisioned
      --kernel-page-cache=true
      --default-log-level=debug
      --smp=1
    State:          Running
      Started:      Sat, 04 Dec 2021 18:42:04 +0700
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  2Gi
    Requests:
      cpu:     1
      memory:  2Gi
    Environment:
      REDPANDA_ENVIRONMENT:  kubernetes
      POD_NAME:              external-connectivity-0 (v1:metadata.name)
      POD_NAMESPACE:         default (v1:metadata.namespace)
      POD_IP:                 (v1:status.podIP)
    Mounts:
      /etc/redpanda from config-dir (rw)
      /var/lib/redpanda/data from datadir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-m8xn5 (ro)
  rpk-status:
    Container ID:  docker://0f9414af80f1669097a6934e7e844d8f8f6922211c900969b5c2c24653c0c3d4
    Image:         vectorized/redpanda:latest
    Image ID:      docker-pullable://vectorized/redpanda@sha256:65531e3c16b313891db67a5c6cf2839569c845840e8884966c64a6d7611d45c7
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/local/bin/rpk-status.sh
    State:          Running
      Started:      Sat, 04 Dec 2021 18:42:08 +0700
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  10M
    Requests:
      cpu:     200m
      memory:  10M
    Environment:
      REDPANDA_ENVIRONMENT:  kubernetes
    Mounts:
      /etc/redpanda from config-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-m8xn5 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  datadir:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  datadir-external-connectivity-0
    ReadOnly:   false
  configmap-dir:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      external-connectivity-base
    Optional:  false
  config-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-m8xn5:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Guaranteed
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                            node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  25m   default-scheduler  0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
  Normal   Scheduled         25m   default-scheduler  Successfully assigned default/external-connectivity-0 to docker-desktop
  Normal   Pulling           25m   kubelet            Pulling image "vectorized/configurator:v21.10.2"
  Normal   Pulled            25m   kubelet            Successfully pulled image "vectorized/configurator:v21.10.2" in 15.307799698s
  Normal   Created           25m   kubelet            Created container redpanda-configurator
  Normal   Started           25m   kubelet            Started container redpanda-configurator
  Normal   Pulling           25m   kubelet            Pulling image "vectorized/redpanda:latest"
  Normal   Pulled            24m   kubelet            Successfully pulled image "vectorized/redpanda:latest" in 50.158874321s
  Normal   Created           24m   kubelet            Created container redpanda
  Normal   Started           24m   kubelet            Started container redpanda
  Normal   Pulling           24m   kubelet            Pulling image "vectorized/redpanda:latest"
  Normal   Pulled            24m   kubelet            Successfully pulled image "vectorized/redpanda:latest" in 4.108830291s
  Normal   Created           24m   kubelet            Created container rpk-status
  Normal   Started           24m   kubelet            Started container rpk-status

$ kubectl describe pod external-connectivity-1

$ kubectl describe pod external-connectivity-1
Name:           external-connectivity-1
Namespace:      default
Priority:       0
Node:           <none>
Labels:         app.kubernetes.io/component=redpanda
                app.kubernetes.io/instance=external-connectivity
                app.kubernetes.io/name=redpanda
                controller-revision-hash=external-connectivity-d95c7cdbc
                statefulset.kubernetes.io/pod-name=external-connectivity-1
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  StatefulSet/external-connectivity
Init Containers:
  redpanda-configurator:
    Image:      vectorized/configurator:v21.10.2
    Port:       <none>
    Host Port:  <none>
    Limits:
      cpu:     1
      memory:  2Gi
    Requests:
      cpu:     1
      memory:  2Gi
    Environment:
      SERVICE_FQDN:                     external-connectivity.default.svc.cluster.local.
      CONFIG_SOURCE_DIR:                /mnt/operator
      CONFIG_DESTINATION:               /etc/redpanda/redpanda.yaml
      REDPANDA_RPC_PORT:                33145
      NODE_NAME:                         (v1:spec.nodeName)
      EXTERNAL_CONNECTIVITY:            true
      EXTERNAL_CONNECTIVITY_SUBDOMAIN:
      HOST_PORT:                        31595
      PROXY_HOST_PORT:                  31924
    Mounts:
      /etc/redpanda from config-dir (rw)
      /mnt/operator from configmap-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xvl2w (ro)
Containers:
  redpanda:
    Image:       vectorized/redpanda:latest
    Ports:       33145/TCP, 9644/TCP, 9092/TCP, 8082/TCP, 9093/TCP, 8083/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP, 0/TCP, 31595/TCP, 31924/TCP
    Command:
      /usr/bin/rpk
    Args:
      redpanda
      start
      --check=false
      --advertise-rpc-addr=$(POD_NAME).external-connectivity.default.svc.cluster.local.:33145
      --overprovisioned
      --kernel-page-cache=true
      --default-log-level=debug
      --smp=1
    Limits:
      cpu:     1
      memory:  2Gi
    Requests:
      cpu:     1
      memory:  2Gi
    Environment:
      REDPANDA_ENVIRONMENT:  kubernetes
      POD_NAME:              external-connectivity-1 (v1:metadata.name)
      POD_NAMESPACE:         default (v1:metadata.namespace)
      POD_IP:                 (v1:status.podIP)
    Mounts:
      /etc/redpanda from config-dir (rw)
      /var/lib/redpanda/data from datadir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xvl2w (ro)
  rpk-status:
    Image:      vectorized/redpanda:latest
    Port:       <none>
    Host Port:  <none>
    Command:
      /usr/local/bin/rpk-status.sh
    Limits:
      cpu:     200m
      memory:  10M
    Requests:
      cpu:     200m
      memory:  10M
    Environment:
      REDPANDA_ENVIRONMENT:  kubernetes
    Mounts:
      /etc/redpanda from config-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xvl2w (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  datadir:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  datadir-external-connectivity-1
    ReadOnly:   false
  configmap-dir:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      external-connectivity-base
    Optional:  false
  config-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-xvl2w:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Guaranteed
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                            node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  26m   default-scheduler  0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
  Warning  FailedScheduling  26m   default-scheduler  0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.
  Warning  FailedScheduling  25m   default-scheduler  0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.

I noticed a few lines in the logs of default-connectivity-0 where host is empty it seems: {host: , port: ...}

...
INFO  2021-12-04 13:34:13,378 [shard 0] redpanda::main - application.cc:329 - redpanda.advertised_kafka_api:{{kafka:{host: external-connectivity-0.external-connectivity.default.svc.cluster.local., port: 9092}}, {kafka-external:{host: , port: 31595}}}- Address of Kafka API published to the clients
...
DEBUG 2021-12-04 13:34:13,507 [shard 0] cluster - members_manager.cc:145 - updating cluster configuration with {{id: 0, kafka_advertised_listeners: {{kafka:{host: external-connectivity-0.external-connectivity.default.svc.cluster.local., port: 9092}}, {kafka-external:{host: , port: 31595}}}, rpc_address: {host: external-connectivity-0.external-connectivity.default.svc.cluster.local., port: 33145}, rack: {nullopt}, properties: {cores 1, mem_available 0, disk_available 0}, membership_state: active}}
...
INFO  2021-12-04 13:34:13,378 [shard 0] redpanda::main - application.cc:329 - pandaproxy.advertised_pandaproxy_api:{{proxy:{host: external-connectivity-0.external-connectivity.default.svc.cluster.local., port: 8082}}, {proxy-external:{host: , port: 31924}}}	- Rest API address and port to publish to client
...
INFO  2021-12-04 13:34:13,465 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:1171 - started raft, log offsets: {start_offset:0, committed_offset:1, committed_offset_term:1, dirty_offset:1, dirty_offset_term:1, last_term_start_offset:0}, term: 1, configuration: {current: {voters: {{id: {0}, revision: {0}}}, learners: {}}, old:{nullopt}, revision: 0, brokers: {{id: 0, kafka_advertised_listeners: {{kafka:{host: external-connectivity-0.external-connectivity.default.svc.cluster.local., port: 9092}}, {kafka-external:{host: , port: 31595}}}, rpc_address: {host: external-connectivity-0.external-connectivity.default.svc.cluster.local., port: 33145}, rack: {nullopt}, properties: {cores 1, mem_available 0, disk_available 0}, membership_state: active}}}

On my Google Cloud Engine logs, I printed the logs there and can verify there is no instance of host: , which makes me extremely sure that this host being empty is the issue. If I can somehow make it 0.0.0.0 instead of being empty, I'm sure it would fix it.

[emaxerrno]

@i-x and @alenkacz could probably help. Also we should fix docs after this

[alenkacz]

This is on our short-term roadmap so should be resolved soon 🤞

[tqwewe]

Thanks @alenkacz , please let me know when there's an update for this 🙏 I need to work with redpanda in kubernetes to continue with my project sadly