Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alertmanager pod is stuck in pending state if the masters are not labeled with "node-role.kubernetes.io/worker" #439

Open
pratik705 opened this issue Sep 13, 2024 · 0 comments
Assignees

Comments

@pratik705
Copy link
Contributor

Describe the bug
Following affinity is set for Alertmanager[1]. Additionally, nodeSelector is also set to the controller[2]. If the controllers/masters are not labeled as node-role.kubernetes.io/worker, Alertmanager will be stuck in a pending state, as expected. To resolve the issue, I had to remove the affinity.
[1]

    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: node-role.kubernetes.io/worker
                  operator: In
                  values:
                    - worker

[2]

      nodeSelector:
        openstack-control-plane: enabled

To Reproduce
Steps to reproduce the behavior:

  1. Dont label the master/controlelrs with node-role.kubernetes.io/worker.
  2. Try deploying the prometheus following the instructions

Current behavior

  • Alertmanager will be stuck in a pending state.
kubectl get pod -A |egrep -i prom
prometheus       alertmanager-kube-prometheus-stack-alertmanager-0           0/2     Pending   0             109s
prometheus       kube-prometheus-stack-kube-state-metrics-7db65fb76b-fjldc   1/1     Running   0             18h
prometheus       kube-prometheus-stack-operator-b87456bcf-k2rnj              1/1     Running   0             18h
prometheus       kube-prometheus-stack-prometheus-node-exporter-5kv9p        1/1     Running   0             18h
prometheus       kube-prometheus-stack-prometheus-node-exporter-7dfj7        1/1     Running   0             18h
prometheus       kube-prometheus-stack-prometheus-node-exporter-bl4rf        1/1     Running   0             18h
prometheus       kube-prometheus-stack-prometheus-node-exporter-f6bct        1/1     Running   0             18h
prometheus       kube-prometheus-stack-prometheus-node-exporter-g9hzj        1/1     Running   0             18h
prometheus       kube-prometheus-stack-prometheus-node-exporter-q8twv        1/1     Running   0             18h
prometheus       prometheus-kube-prometheus-stack-prometheus-0               2/2     Running   0             6m44s

Expected behavior

  • Alertmanager should be in running state.
@pratik705 pratik705 self-assigned this Sep 13, 2024
LukeRepko pushed a commit that referenced this issue Sep 16, 2024
Handle Alertmanager installation using `nodeSelector`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant