Preview local changes with flux diff kustomization

[stefanprodan]

Followup of: fluxcd/kustomize-controller#426

To allow users to preview changes to their Kustomize overlays without committing changes to upstream, Flux CLI could offer two commands:

• flux build kustomization my-app -f ./path/to/local/manifests
• flux diff kustomization my-app -f ./path/to/local/manifests

The build command queries the Kubernetes API and fetches the specified Flux Kustomization, then it uses the specified local path -f to build the overlay (using the same logic as kustomize-controller) to write the resulting multi-doc YAML to stdout.

The diff command does the same thing as build but instead of outputting the manifest, it uses them to perform a server-side dry-run and write the YAML diff to stdout. An example of how the output could look can be found here: https:/stefanprodan/kustomizer.

Given that SOPS encrypted secrets can't be decrypted outside the cluster (due to IRSA, Azure AD, etc) the build and diff commands should detect the SOPS secrets and skip them from the output.

We need to extract the kustomize patch overrides, kustomize build and post build variable substitution logic from kustomize-controller into fluxcd/pkg then use these helpers in both the controller and Flux CLI.

[stefanprodan]

@souleb I think that for a PoC of this feature, we could copy the code from kustomize-controller in flux2/internal and implement flux build kustomization, then people could pipe the output to kubectl diff like so:

flux build kustomization podinfo -f ./deploy | kubectl diff -f-

[marcbachmann]

At the moment I'm using kustomize build | node filter-encrypted-secrets.js | kubectl diff -f -
Which works quite well using the following script:

filter-encrypted-secrets.js

#!/usr/bin/env node
function readStdin() {
  process.stdin.setEncoding('utf8')

  return new Promise((resolve, reject) => {
    let string = ''
    process.stdin.on('data', (chunk) => { string += chunk })
    process.stdin.on('end', () => resolve(string))
  })
}

if (!process.stdin.isTTY) {
  (async function exec () {
    const yaml = await readStdin()
    const docs = yaml.split(/^---+/m)
    for (let i = 0; i < docs.length; i++) {
      if (docs[i].match(/^sops:$/m)) continue
      process.stdout.write(docs[i].startsWith('\n') ? `---${docs[i]}` : `---\n${docs[i]}`)
    }
  })()
} else {
  process.stderr.write('Please pipe into this command\n')
  process.exit(1)
}

I also tried to decrypt the secrets using sops instead of filtering them out, but sadly kustomize build reorders the keys (even with --reorder none) and then the sops hmac verification fails. Would be neat to have that somehow working even with a local setup.

[souleb]

@marcbachmann thanks for the example. For now, sops encrypted secret valued will not be used for diff. We will diff on the keys only. For not encrypted secrets, we will have a full diff.

Any thoughts on how you would decrypt with a local setup?

[marcbachmann]

I guess a clean solution would be to wrap filesys and override the ReadFile function to read the secret and directly decrypt it.
Or for a kustomize build | sops -d pipeline (without ReadFile update), a PR against the sops hmac verification logic could also be done. It would need to ignore the yaml property order. But decrypting after kustomize build might break on encrypted secrets in case the passthrough behavior changes in kustomize. And then the sops team first needs to accept such a change.

Any thoughts on how you would decrypt with a local setup?

Doesn't the sops module already properly handle the key lookup using env variables and default locations?
I'm usually using age keys at the default key location or aws-vault to set up the aws profile that has the kms key present.

It's definitely good to first implement the support without encrypted secret support, as this is the most common case. Most likely not all developers working in a flux setup have set up the keys.

Thanks for working on that 👏

[jrauschenbusch]

@stefanprodan How would the output of flux diff differ from the output of kustomizer diff?

One example: I tried to implement a diff based on a feature branch workflow where a CI pipeline job runs on pull / merge requests and reports the changes that would be applied to the cluster, as soon as the code changes are merged. I chose dyff for this job by using the KUBECTL_EXTERNAL_DIFF option. The problem with this is currently that the metadata labels / annotations added by the flux controller are also reported. That makes it difficult to analyze the report. Would this be addressed by flux diff. The nice thing with dyff is that you can see the changes on YAML path level per resource. But the output of kustomizer diff is also quite nice.

[stefanprodan]

@jrauschenbusch flux diff should run the same code as kustomize-controller, this means all labels, patches and other transformations should be the same.

[jrauschenbusch]

@stefanprodan So if i get you right, then this added metadata / annotation shouldn't be reflected in the diff output, right?

[stefanprodan]

this added metadata / annotation shouldn't be reflected in the diff output, right?

There will be no diff to show since the labels added by the controller will be the same with the ones added by the CLI.