Upgrade to 2.2.3 not solve the issue: kustomization.yaml is empty

[khoinguyen]

Describe the bug

I upgraded to 2.2.3, but still get the same error (as described and closed in #4509):
Logs:

kc logs kustomize-controller-6464685c8c-p9tdr | grep empty
{"level":"error","ts":"2024-02-12T05:55:53.572Z","msg":"Reconciliation failed after 10.825548072s, next try in 30m0s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"infra-unstable","namespace":"flux-system"},"namespace":"flux-system","name":"infra-unstable","reconcileID":"1ba7665d-f7d2-49ce-a6fe-2dcb1ad63725","revision":"develop@sha1:91026b76153cf514fd4ad29d3c88988cf6f91789","error":"kustomize build failed: kustomization.yaml is empty"}
{"level":"error","ts":"2024-02-12T05:56:07.288Z","msg":"Reconciliation failed after 10.707927756s, next try in 30m0s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"apps-unstable","namespace":"flux-system"},"namespace":"flux-system","name":"apps-unstable","reconcileID":"c0b921ab-72d8-408c-b97f-cf9fd54c208d","revision":"develop@sha1:91026b76153cf514fd4ad29d3c88988cf6f91789","error":"kustomize build failed: kustomization.yaml is empty"}
{"level":"error","ts":"2024-02-12T05:56:09.102Z","msg":"Reconciliation failed after 8.76190182s, next try in 30m0s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"infra-unstable","namespace":"flux-system"},"namespace":"flux-system","name":"infra-unstable","reconcileID":"db728136-7afc-48f9-9d16-61b199ec24f5","revision":"develop@sha1:91026b76153cf514fd4ad29d3c88988cf6f91789","error":"kustomize build failed: kustomization.yaml is empty"}
{"level":"error","ts":"2024-02-12T05:56:23.465Z","msg":"Reconciliation failed after 4.662710228s, next try in 30m0s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"apps-unstable","namespace":"flux-system"},"namespace":"flux-system","name":"apps-unstable","reconcileID":"4d9d3065-8f50-4094-ac28-ceb2556d968c","revision":"develop@sha1:91026b76153cf514fd4ad29d3c88988cf6f91789","error":"kustomize build failed: kustomization.yaml is empty"}
{"level":"error","ts":"2024-02-12T06:02:03.073Z","msg":"Reconciliation failed after 9.98762563s, next try in 30m0s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"apps-unstable","namespace":"flux-system"},"namespace":"flux-system","name":"apps-unstable","reconcileID":"c1214416-d58a-45ec-8d83-1b5d643afbf4","revision":"develop@sha1:691798195af8f47ba6254ddcc36b5fec7bb1cc45","error":"kustomize build failed: kustomization.yaml is empty"}
{"level":"error","ts":"2024-02-12T06:02:03.073Z","msg":"Reconciliation failed after 9.987724185s, next try in 30m0s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"infra-unstable","namespace":"flux-system"},"namespace":"flux-system","name":"infra-unstable","reconcileID":"9d7b9d04-65fe-4364-a535-1993bc4e7270","revision":"develop@sha1:691798195af8f47ba6254ddcc36b5fec7bb1cc45","error":"kustomize build failed: kustomization.yaml is empty"}
{"level":"error","ts":"2024-02-12T06:03:34.776Z","msg":"Reconciliation failed after 9.789412089s, next try in 30m0s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"apps-unstable","namespace":"flux-system"},"namespace":"flux-system","name":"apps-unstable","reconcileID":"85e4c6a5-f37e-408b-81fc-4381373cecc4","revision":"develop@sha1:8883bf3d29b13eb5c98147078d3bed26b75efa82","error":"kustomize build failed: kustomization.yaml is empty"}
{"level":"error","ts":"2024-02-12T06:03:34.877Z","msg":"Reconciliation failed after 9.891731856s, next try in 30m0s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"infra-unstable","namespace":"flux-system"},"namespace":"flux-system","name":"infra-unstable","reconcileID":"3ff90165-8b87-43a2-adaa-8744bfbb1fb9","revision":"develop@sha1:8883bf3d29b13eb5c98147078d3bed26b75efa82","error":"kustomize build failed: kustomization.yaml is empty"}

Basically too avoid blocking in our system, the new charts or apps will be in *-unstable root kustomization, when it is stable, we move it to *-stable kustomization. This renders most of the time the *-unstable kustomization are empty.

Steps to reproduce

1. Upgrade to FluxCD 2.2.3, kustomize-controller upgraded to 1.2.2
2. Have following Kustomization definition:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infra-unstable
  namespace: flux-system
spec:
  interval: 30m0s
  path: ./infrastructure/kubernetes/infra/unstable
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

and

apiVersion: kustomize.config.k8s.io/v1
kind: Kustomization
resources: []

3. Wait for FluxCD to changes

Expected behavior

Just silently skip the empty Kustomization

Screenshots and recordings

No response

OS / Distro

macOS sonoma (client)

Flux version

v2.2.3

Flux check

flux check
► checking prerequisites
✔ Kubernetes 1.26.12-eks-5e0fdde >=1.26.0-0
► checking version in cluster
✔ distribution: flux-v2.2.3
✔ bootstrapped: true
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.37.4
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.37.1
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.31.2
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v1.2.2
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v1.2.4
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v1.2.4
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta3
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta2
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ imagepolicies.image.toolkit.fluxcd.io/v1beta2
✔ imagerepositories.image.toolkit.fluxcd.io/v1beta2
✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta3
✔ receivers.notification.toolkit.fluxcd.io/v1
✔ all checks passed

Git provider

github

Container Registry provider

ECR

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[jtucci]

Running into this issue too.

[stefanprodan]

What we fixed in Flux is when you have no kustomization.yaml. If you do use that file, you need to add namespace: <something> and it will not error out.

[gracedo]

@stefanprodan Just to clarify, an "empty" kustomization.yaml file (no resources) now requires a namespace field? Non-empty kustomization.yaml files don't require one, what is the explanation behind this new requirement being introduced only for "empty" ones?

Empty kustomization.yaml being:

apiVersion: kustomize.config.k8s.io/v1
kind: Kustomization
resources: []

[stefanprodan]

Here is the best solution to workaround the breaking change in kustomize, without affecting the generated objects later on.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
buildMetadata: [ originAnnotations ]
resources: []

Adding originAnnotations also allows for resources: [] to be omitted.

@gracedo the reason is that the latest kustomize version doesn't allow empty kustomization anymore, see kubernetes-sigs/kustomize#4949

[stefanprodan]

The fix for this breaking change will be included in Flux 2.3