Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Helm-controller doesn't restore Custom Resource after helm release failed #740

Open
ginn13 opened this issue Jul 21, 2023 · 0 comments
Open

Comments

@ginn13
Copy link

ginn13 commented Jul 21, 2023

Helm-controller v0.35.0 (also checked on v0.31.2)
Example CR HR

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: my-frontend
  namespace: default
spec:
  chart:
    spec:
      chart: my-frontend
      reconcileStrategy: ChartVersion
      sourceRef:
        kind: HelmRepository
        name: myrepo
        namespace: default
      version: 2.3.1
  interval: 1m
  releaseName: my-frontend
  timeout: 15m
  values:
    ingress:
      enabled: true
      hostname: service.com
    customResource:
      enabled: true
      frontend:
        client:
          rootUrl: https://service.com

Say we have 2 resources in helm chart.
one Ingress (state A)
one CustomResource KeycloakClient (state A)

If I change Ingress declaration from correct to incorrect, for example bad hostname. And also make change to CustomResource, for example some value rootUrl.
Say Ingress goes state B. CustomResource goes state B.

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: my-frontend
  namespace: default
spec:
  chart:
    spec:
      chart: my-frontend
      reconcileStrategy: ChartVersion
      sourceRef:
        kind: HelmRepository
        name: myrepo
        namespace: default
      version: 2.3.1
  interval: 1m
  releaseName: my-frontend
  timeout: 15m
  values:
    ingress:
      enabled: true
      hostname: service.
    keycloakClient:
      enabled: true
      frontend:
        client:
          rootUrl: https://service.

Then HR becomes Failed and I get error

Helm upgrade failed: cannot patch "my-frontend" with kind Ingress: Ingress.extensions "my-frontend" is invalid: spec.rules[0].host: Invalid value: "service.": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*') Last Helm logs: preparing upgrade for my-frontend resetting values to the chart's original version performing update for my-frontend creating upgraded release for my-frontend warning: Upgrade "my-frontend" failed: cannot patch "my-frontend" with kind Ingress: Ingress.extensions "my-frontend" is invalid: spec.rules[0].host: Invalid value: "my.": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')

If I rollback changes to the HR, then the HR becomes Ready.
Ingress comes back to state A.
CR KeycloakClient stay unchanged and keeps in state B.

If I reproduce steps on just helm (v3.12.1) only:

helm upgrade --install my-frontend-2.3.1.tgz -f brokenValues.yaml

Helm release becomes Failed. Ingress goes state B, CR KeycloakClient goes state B.
After applying previous correct values:

helm upgrade --install my-frontend-2.3.1.tgz -f correctValues.yaml

Helm release becomes Deployed. Ingress goes back to state A, CR KeycloakClient goes back to state A.

It seems that helm-controller doesn't change the CR after upgrading release from failed state. If target declaration of CR is equal to last working state of helm release.

Current workaround is to use remediateLastFailure: true

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
spec:
  upgrade:
    remediation:
      remediateLastFailure: true

to just not to break CustomResource while breaking the release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant