Correct documentation about SOPS for different file types than YAML

[quaideman]

Issue

Upgrading from Flux v0.28.5 to v0.30.2 (kustomize-controller v0.23.5) using the bootstrap method recreated all of the secrets in our cluster, breaking SOPS Kustomize generated secrets.

For example, before the upgrade (kustomize-controller v0.22.3) the secret would look like this after decoding the data field:

{
  "auths": {
    "quay.io": {
      "auth": "XXX",
      "email": ""
    }
  }
}

And after the upgrade (kustomize-controller v0.23.5):

{
        "data": "{\n  \"auths\": {\n    \"quay.io\": {\n      \"auth\": \"XXX\",\n      \"email\": \"\"\n    }\n  }\n}"
}

Forcing the kustomize-controller back to version v0.22.3 (leaving everything else) triggered all the secrets to get recreated again and reverted them back to a working state.

Replicate issue:

creds.txt:

{
  "auths": {
    "quay.io": {
      "auth": "XXX",
      "email": ""
    }
  }
}

Encrypt the file:
sops -p "fingeprint" -e creds.txt > creds.json

creds.json:

{
	"data": "ENC[AES256_GCM,data:FrgCy1umHiEx9+D7J/xPNfZhaBxbET5LFGLQUp9VbO+Z+OuM3Xa+0CMKinJuRt/aDhKQeCMkjiSJMU1aLEihbmqLa96nJsTfWKjb4aUV8rMu1g==,iv:NfNf0PLzrM7n+P7WbGEd0XvxzyK0HWatfB4xrVlw/Cc=,tag:g69YGCoWhGyMJO9rAqWPFw==,type:str]",
	"sops": {
		"kms": null,
		"gcp_kms": null,
		"azure_kv": null,
		"hc_vault": null,
		"age": null,
		"lastmodified": "2022-05-18T16:21:12Z",
		"mac": "ENC[AES256_GCM,data:BF5pJb2h4OggiG8d4UVBLRn8kBLqeySEQBrG4rEOE7Ars8U6TFqXXj3qgU3+/dmCjJ4hhKAKyuDOi2RvwDkzAEZORn0vToxzmbqpuIcbdQdHmpub74dlLclTZZyorDdJVCfwmkBbEsDq9ciW3JM5/gP8pZH+6vGJFiBNHENyIcQ=,iv:hw+AQMNkfiu1ASqh7xns9Mw5WK6/OUW9SYgUa8+rrJ0=,tag:ZGCRgmjxW0Yl4Tjsa/fpHA==,type:str]",
		"pgp": [
			{
				"created_at": "2022-05-18T16:21:12Z",
				"enc": "-----BEGIN PGP MESSAGE-----\n\nhQIMA5RBFvcCBP9tARAAi0+vEDKdENjCyqj9ImPXSXf5JHrOSpaRkDS+Cg9Ir+O4\nvlifuN5ZHYQ7Pt+JNBAp+VAAEySx9Xow82TKZ6wI1z3hJ58Cj35wNJ8Z3ktED/Zj\nGUx8K1B1IvfcjidOteljZyGo8wmFRj5MuiHIPn0Ub60dpL7yZ5d04c+XAAJUltKA\ngcfufRN3sM0ifdz6hYsstkwLkQbP4HMlnGLoTH+E94A4w578/fssxQC2MUuzPHrI\nFLXI6ykwMfibQuaXV+3vJLI2oq1FR7KuOWLV77zPqIT15/7g1B/a3pq9oTcCpbUU\nsMuRs25kY0ttGZyIsycbYjx1Vg0F/GrNgUNfE5IEojNpnwmuO/7IvQIkLoqHRIvG\nmPpMWGsHOevG5ktZnbMd1HlHtKhRiv4nMvpfoh/PaRAvIT86MT3xs39UDce6YAC0\nzHCYz3Z7tGssWESkg48QMXLGxZLiZEtwu1QOCuHs91nWIijUgO9Fkb6Qj7l+6VXT\nTcCs2sr063WP7Suvh6rDJu6U/cJvnM3z1wI1HU+RNg9CBW4TzKojj9o/JqvPJqB9\ncF8X7ejrdVQm8fBF/jkMoZicqs+EAjbE+RQnDQzY/C3M8iy952LJvYTSGVxZfNQ0\nsD3dVSqLY/IzZ9GJ2hWwhJsu5PHVII7uVy6aBDn8RbkErS8h4wa9bmqKYnBWW8PS\nXgHm7COw8IElO1Lxg2nO9nGUDwjIKFbKijVyXzTTWU7+ULFnNj1m9/YC0Nu3LGOD\nx2Q6nTf0ocNaj9MBMhRP77PACvnnLDTpFoM7AKkCsbebpftB3KNHIDO+plEth8g=\n=zS/2\n-----END PGP MESSAGE-----\n",
				"fp": "76AFF750D2F059316D7B430DA0298AE04F92D209"
			}
		],
		"unencrypted_suffix": "_unencrypted",
		"version": "3.7.2"
	}
}

kustomization.yaml

secretGenerator:
- name: quay.io
  type: kubernetes.io/dockerconfigjson
  options:
    disableNameSuffixHash: true
  files:
  - .dockerconfigjson=secrets/creds.json

[kingdonb]

I didn't test the pre-upgrade behavior out yet, but I can confirm the generated secret comes out like this:

{
        "data": "{\n  \"auths\": {\n    \"quay.io\": {\n      \"auth\": \"XXX\",\n      \"email\": \"\"\n    }\n  }\n}"
}

The content of the json file is encapsulated in a data field, which it apparently never was before. Is this a regression?

I did not notice this at first (we were discussing this in Slack, trying to resolve it there), but in the docs, it is mentioned that you should mark up your sops encrypting with an --input-type if you are encrypting anything but the garden path of only data fields in a YAML file (types can be json, env, or yaml to encrypt a whole file).

It didn't seem to fix anything at first, but on a second glance it looks like adding the --input-type=json as described in the docs really does have an effect. This is mentioned in the docs here:

https://fluxcd.io/docs/components/kustomize/kustomization/#kustomize-secretgenerator

With the input-type=json, I get this result (it looks normal):

$ base64 -D
ewoJImF1dGhzIjogewoJCSJxdWF5LmlvIjogewoJCQkiYXV0aCI6ICJYWFgiLAoJCQkiZW1haWwiOiAiIgoJCX0KCX0KfQ==
^D
{
	"auths": {
		"quay.io": {
			"auth": "XXX",
			"email": ""
		}
	}
}

and without it, I get the json encapsulated in a "data" field as shown above.

I do not know what would have changed between 0.28 and 0.29 to cause this, I've been through the changelogs in Kustomize controller and I see there were some SOPS changes that most certainly are relevant in the 0.24.0 release of Kustomize Controller which landed in Flux 0.29.0:

https:/fluxcd/kustomize-controller/blob/v0.24.0/CHANGELOG.md#0240

I wouldn't rule out that this could be a regression, as I don't see it mentioned as a breaking change at the top level, it only gets this mention, not listed in the section marked for breaking changes:

SOPS refactored and optimized
SOPS implementation was refactored to include various improvements and extended code coverage. Age identities are now imported once and reused multiple times, optimizing CPU and memory usage between decryption operations.

I suspect this PR was the one that had the impact:

• controllers: improve decryptor and add tests #619

[kingdonb]

At a guess the old behavior was selecting the JSON format (based on the filename?), and the new behavior selects the binary format instead by default:

https:/fluxcd/kustomize-controller/pull/619/files#diff-efcb6cc0bdb62e54699cd11dfb9fbbab9f2ae76ada019846e853667ab29c86b0R79

(which would explain why JSON format data is being stored in a "data" element rather than regurgitated as plain JSON.)

[bigfleet]

Dropping by to note that I'm observing the same behavior.

bigfleet@jarvis:~/lvvl/eagle/ansible/flux2-builder$ yq e ".data[.dockerconfigjson]" tmp/clusters/config/pull-secret.yaml | base64 --decode 
{
        "data": "ENC[AES256_GCM,data:XXqpZUg4Dy+MzNAu9ODjuw+YNim0yK6M0CD3P+mJPDPrqelqHoXC3EBFfewNgCMQHOE/yj+RFvmknSQR+dJ4hKrKXckkTc4JJtXzjB4YVAFakQ2Q2eohaDIfwsFyxE+s7iUQOg8gzQD5Tx2G/PtNcQfvHy5Qrw428+dykkFmDOKP1rBW+usyQnPqw2mh3F/qqvxc3RJaxEfL51v+KcqaevzekSA1knQkUAxWfBACOn3+P7Yvwlm7CWgefUodrFs1d6H9T6XklVBUI0kDU1C3OZaOF11wbECph5FsGmzspMEMXkNP5J45eZIMmNdedFWdGd8N1FXMtu3PSUXnRbRILvvVs2eInN8pQXMIsRunGBY=,iv:nmUWxNTBim/OZD1Y0y8iRgY17H4PPaKCWLst6rcWGcs=,tag:TAbLjGLx6Ocb3Xh29mrkXg==,type:str]",
        "sops": {
                "kms": null,
                "gcp_kms": null,
                "azure_kv": [
                        {
                                "vault_url": "https://avaultname.vault.azure.net",
                                "name": "sops-key",
                                "version": "ae97b7c9987e40249780290dab967530",
                                "created_at": "2022-05-10T01:31:00Z",
                                "enc": "NE0o8wbT3tudHgEkx_jiliq7dzsmmTawxio28IdDd5Tc90u2TQXGRxfEGO4nnZroTbX0MwsnFkthNPqXTDBR2_BaUspg7_jOb0qgkhuA9Sk5H4pj1mlfxr9-WfFHcRoFcUmC_4PKb74WnrbxwEH5X10nDEJ1JeMTNgMq2zM3PMle6bvxyiax0wIYQxF5-aSab9Q3a4jz3FyB_KN7ZYfeZxtcEtcAIXVc3XfnM9xP8nN8Zz9BDENCsFELsv9g1LcpPYVu1Qb-ZOft5ooinsaKgNDIwmXGovjL4mSKewcZkNBq4hO8Iy6kF8gV12hn3OGMvZ1aRAi__H9CESTFbTO6dA"
                        }
                ],
                "hc_vault": null,
                "age": null,
                "lastmodified": "2022-05-10T01:31:01Z",
                "mac": "ENC[AES256_GCM,data:olX/fOrwvucBop+3aFVfMwg7tL9L4PrsMgvkw5doh90h5BRJ9wYAZlVE60/1bmpx71Tlsp8gVEEprXHqOHRxyLV4cWQZ1kNOfiw2WX1qpuynIiXF4qyWC7JzHZNSFzSXeuS5Y9TmS3e/qapU7knJGrfMvaUIaX2yu2nC3iFHxZw=,iv:/GMcTTap+jA82Xsb7R/UK3VpJDKmA2CNpGsIf1ZxevQ=,tag:xr6AhmQORahtT2e1tVS24g==,type:str]",
                "pgp": null,
                "unencrypted_suffix": "_unencrypted",
                "version": "3.7.1"
        }
}

Writing that to disk and processing the output:

bigfleet@jarvis:~/lvvl/eagle/ansible/flux2-builder$sops --decrypt --azure-kv ..... tst.out | jq
{
  "auths": {
    "nameofrepo.azurecr.io": {
      "username": "ausernamebaby",
      "password": "garbageelided",
      "auth": "LONGHEXADECIMALBADBEEF..."
    }
  }
}

So the "new" behavior is happening when the sops encrypted material is being extracted from the contents (rather than an issue on the way "into" the flux repo) in this particular case. (I can provide more details about my case if it's helpful, certainly.)

[hiddeco]

What version of the controller are you on? I have a feeling this was indeed a regression, but solved by #644 which is available in v0.25.0.

[bigfleet]

Attempted from scratch just now. Unfortunately no change, and it did report 0.25 for kustomize controller.

bigfleet@jarvis:~/lvvl/eagle/ansible$ kubectl describe -n flux-system deployment/kustomize-controller | grep Image
    Image:       ghcr.io/fluxcd/kustomize-controller:v0.25.0

bigfleet@jarvis:~/lvvl/eagle/ansible$ kubectl get -n flux-system secret flux2-pull-secret -o json | jq -r '.data[".dockerconfigjson"]' | base64 -d -
{
        "data": "{\"auths\": {\"name.azurecr.io\": {\"username\": \"badbad0-babe-baa\", \"password\": \"password-elided\"}}}"

[stefanprodan]

@bigfleet did you pass --input-type=json to the SOPS CLI at encryption time?

[bigfleet]

@stefanprodan I was in position to alter the handling of the encrypted data to provide the input type hint, and it did resolve the issue. So, for this participant, there is a "workaround" by being more explicit with SOPS regarding the input you're encrypting.

I will leave to others whether there's something to be done here or not. Behavior has changed for default params-- is it ever a breaking change if there was no behavior committed to? Not for me to say.

[kingdonb]

Maybe this can be closed? The behavior should be documented, although it is already documented how to use this, I'm not sure what changed was inside Flux. It sounds like you tested and found the behavior has changed in Flux, based on a SOPS upgrade from https:/fluxcd/kustomize-controller/blob/v0.24.0/CHANGELOG.md#0240 it might have changed.

If the issue can be solved by being a bit more explicit in the pipeline, can we commit to the current behavior now, or are we at the mercy of SOPS upstream if it changes again?

I don't know but I think this can be closed.

[bigfleet]

I would be in favor of closing in preference of a ticket that updates the docs to provide a commented example of how to get the desired results. (This will be a common use case, enabling flux to leverage pull secrets.)

I don't think any current docs include the input-type option to sops, or explain the importance of their correct usage. I can help write them if this approach sounds correct.

[hiddeco]

I have asked @seh for some input, as he was really helpful in providing scenarios and reasoning around the patch I made in #644. This won't be before tomorrow however.

[seh]

I agree that the whole way that SOPS accepts input and output types is confusing. You can get away with just telling it that everything is "binary," and if you control both the encryption and decryption sides, it all works fine. The trouble comes when Flux has to apply heuristics to guess what the input type had been and to choose an output type.

Hidde's #644 addressed a complaint of mine: When I encrypted a JSON-formatted Docker credential and I told SOPS that the input type was JSON, Flux didn't believe me because the Secret data key is ".dockerconfigjson"—as mandated by the Kubernetes API— which does not end in the ".json" sentinel suffix. (Note the missing period.)

If I "lied" to SOPS and told it that my input data was "binary" instead of JSON, then when Flux got it wrong and decrypted this Secret field as "binary," I got back out what I had put in originally. However, I then had to know that I had to "lie" to SOPS, which seemed unfair. Hidde corrected Flux's heuristic in this case, so that it would guess correctly that the input type had most likely been JSON.

As Jim suggested in #654 (comment), I think the current behavior is optimal, but that we'd all benefit from clarifying documentation that tells users exactly how to prepare a JSON-formatted Docker credential for distribution via Flux.

[kingdonb]

I went back and tested against the docs here:

https://fluxcd.io/docs/components/kustomize/kustomization/#kustomize-secretgenerator

They do not seem to work as written, there are two issues that as far as I can tell are not SOPS regressions, as they fail in similar ways on SOPS 3.7.1 (the version from one year ago)

Let's convert this to a Docs issue, as long as some folks are using this successfully we can be reasonably assured that the functionality overall works, but I am less confident the docs are correct.

These statements in particular:

cat config.json | sops -e --input-type=json > config.json.encrypted
cat config.yaml | sops -e --input-type=yaml > config.yaml.encrypted
cat config.env | sops -e --input-type=env > config.env.encrypted

I'm not sure how these commands should work. SOPS apparently requires you to provide an input file, it will not accept - and it will not default to STDIN, you have to tell it /dev/stdin if you want it to encrypt from stdin, so these commands all error out like this:

Error: no file specified

But even with stdin, these commands all depend on your .sops.yaml to tell SOPS how to encrypt the data and to whom. There must be a creation rule that matches the path. Else you will get an error like:

config file not found and no keys provided through command line options
or
error loading config: no matching creation rules found

I added these in mine:

creation_rules:
  - path_regex: .*.yaml
    encrypted_regex: ^(data|stringData)$
    pgp: A883B02E14B7EAF0A6BDAF69ECD23C5916E31D4F
  - path_regex: .*.env
    pgp: A883B02E14B7EAF0A6BDAF69ECD23C5916E31D4F
#  - path_regex: .*.json
#    pgp: A883B02E14B7EAF0A6BDAF69ECD23C5916E31D4F
#  - path_regex: dev\/stdin
#    pgp: A883B02E14B7EAF0A6BDAF69ECD23C5916E31D4F

two rules at the bottom, (commented out because I thought they should not be necessary and I think they do not come from our docs anywhere.) These commands start working with the additions uncommented to match these other paths.

I am at a loss why I missed this when I tried last week. These are all SOPS troubles though, and the creation of secrets from Flux's perspective is a SOPS responsibility, so we can fix it through docs, and we can try to get something fixed in SOPS if the behavior is wrong, but on Flux's side I think this is just a documentation issue, I'll change the title to reflect that.

[kingdonb]

(I do not have write access to the repo so I cannot update titles)

/rename Correct documentation about SOPS for different file types than YAML