GnuPG2 2.2 vs 2.1 conflicts in keybox format

[gthank]

What are the steps to reproduce this issue?

1. Initialize git-secret on a machine using gnupg2 version 2.2
2. Use git-secret (tell, add, hide, etc.) and publish the changes
3. Clone your git repo onto a machine using gnupg2 version 2.1 (w/ the proper keys imported)
4. git secret reveal

What happens?

gpg: skipped packet of type 12 in keybox
gpg: skipped packet of type 12 in keybox
gpg: skipped packet of type 12 in keybox
gpg: skipped packet of type 12 in keybox
gpg: skipped packet of type 12 in keybox
gpg: skipped packet of type 12 in keybox
gpg: skipped packet of type 12 in keybox
gpg: skipped packet of type 12 in keybox
gpg: skipped packet of type 12 in keybox
gpg: skipped packet of type 12 in keybox
gpg: skipped packet of type 12 in keybox
gpg: skipped packet of type 12 in keybox

And no secrets are revealed.

What were you expecting to happen?

The secrets to be revealed.

Any other comments?

After digging around, it seems that GnuPG 2.2 added some things to the keybox format that older versions don't have. I realize this is a result of problems with underlying libraries. What I'm hoping is there's some way for you to force an interoperable format by specifying a flag when creating the keybox (or just force use of the older .gpg keyring format instead of the newer .kbx).

What versions of software are you using?

Operating system:

• Darwin hank-mbpr 17.4.0 Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64 x86_64
• Linux falcon 4.4.0-43-Microsoft #1-Microsoft Wed Dec 31 14:42:53 PST 2014 x86_64 x86_64 x86_64 GNU/Linux

git-secret path:

*/usr/local/bin/git-secret

• /usr/bin/git-secret

git-secret version:

• 0.2.2
• 0.2.2

git version:

• git version 2.16.2
• git version 2.7.4

Shell type and version:

• zsh 5.4.2 (x86_64-apple-darwin17.3.0)
• GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)

gpg version:

• gpg (GnuPG/MacGPG2) 2.2.3
• gpg (GnuPG) 2.1.11

[sobolevn]

@gthank thanks for the report. Could you please try the same with the version from master?

[gthank]

@sobolevn I get the same behavior from master (on both the machine creating the keyring and on the machine reading it). I did a sanity check by having the older version of GPG do a --list-keys on the newer keybox version, and it does pull out the info, it just spews the warnings about the skipped packets. However, the secrets are still not actually revealed.

[joshrabinowitz]

@gthank I'd like to try to replicate this. What OS/distribution uses gpg 2.1?
(edit). Oh, I see it's on a Windows installation.

[joshrabinowitz]

I'm starting to think that incompatibilities between different versions of gpg are outside the scope of what git-secret should handle.

I think we should just document the possibility of problems interoperating between GPG versions and close this (unless it's specifically related to git-secret)

Thoughts @gthank @sobolevn ?

[joshrabinowitz]

Perhaps related:
StackExchange/blackbox#222
https://lists.gnupg.org/pipermail/gnupg-devel/2017-May/032846.html

[gthank]

@joshrabinowitz That link is basically exactly what's going on.

Ubuntu 16.04 uses the 2.1 line of code (probably with some backports, given their Debian heritage), which is also how it's a problem on Windows: the WLS thing on Windows uses Ubuntu 16.04.

As for whether to even address this from git-secret: they're just warnings, so the underlying tool does work with different versions. I think the issue is that the unexpected output (probably on stderr?) is breaking git-secret's expectations. Ideally, there would be a simple way to force gnupg to just use the old format, but that's probably not the case. Barring that, would it be possible to just get git-secret to ignore the unexpected output? I'm working around the issue for now by doing all the manipulations of the keyring on the server (an Ubuntu 16.04 box), but that's pretty annoying in the long-term, because it means I have to do the keyring manipulations for all the projects at work, when ideally the lead devs for each project would be able to manage it on their local dev machines: pretty much all Macs running GPG Suite.

[joshrabinowitz]

@gthank I don't think the issue is the unexpected 'gpg: skipped packet of type 12 in keybox' warnings, but rather that gpg returns a non-0 exit code when it returns, which signals error.

[joshrabinowitz]

This issue is described in reasonable detail here: keybase/client#6923

[joshrabinowitz]

@gthank @sobolevn @notjames

#170 Displays an error message and exits if it is unable to decrypt any file.

It also gives a more detailed error message if gpg -n --list-keys returns a non-zero exit code.

I'm not sure what is the ideal behavior in this case. Here are some possibilities:

1. Leave this PR as it is - if we see an exit code from gpg -n --list-keys, show an error and exit.

2. ignore exit codes from gpg -n --list-keys as long as the command appeared to work (IE, output info seems as expected)

3. Something else

What's the ideal solution for git-secret to handle this situation?

Also would be good to create a test for this issue.

[gthank]

@joshrabinowitz @sobolevn @notjames From my perspective, there's not an ideal solution to this from the git-secret side of things: upstream broke something they shouldn't have. The least bad solution, imo, is to ignore the most specific error code you can ignore so long as the command seems to have worked, and then maybe issue a warning on STDERR. It's never fun to have workarounds in your code for weird stuff like this, but I'd say that was the best bet at working for something that's likely to be a fairly common failure mode without accidentally ignoring real problems.

[joshrabinowitz]

@gthank @sobolevn

So, is this the desired logic?

• capture the exit code, stderr and stdout of the gpg -n --list-keys command,
• if captured output consists just of lines like gpg: skipped packet of type 12 in keybox
• then output warning about the issue (without exiting, regardless of exit code)
• check that it looks like we got a list of keys as expected
• if we did not, issue error and exit

[gthank]

@joshrabinowitz @sobolevn I like that idea. Depending on whether gpg -n --list-keys returns a unique error code for this specific case, it may be possible to just check for that instead of parsing stdout/stderr (since parsing that kind of thing is always a hassle), but I have a suspicion it will need to be the stdout/stderr approach.

[joshrabinowitz]

@gthank To follow up -- it does not return a unique exit code. I looked in the source, and as https://unix.stackexchange.com/questions/50541/what-does-gpg-error-code-2gpg-err-unknown-packet-mean explains,

Largely used GPF codes are of three types :
    0 is success (all other values indicate a failure).
    1 for things like a BAD signature.
    2 is usually used for unxpected errors.

[joshrabinowitz]

Thanks for your detailed responses.

For the record, I just successfully tested git-secret inter-operation between

• ubuntu 14 / gpg2 2.1.11,
• OSX / gpg 2.2.6, and
• centos7 / gpg 2.0.22.

I didn't do exhaustive testing, but I was unable to trigger any problems after several 'tell', 'killperson', 'reveal' and 'hide' operations between the machines (with git commits/pushes/pulls as appropriate).

EDIT: I was able to eventually see problems interoperating between the above versions of gpg with git-secret

From my reading, it seems like the solution is to make sure you use interoperating versions of gpg.

If your gpg versions interoperate well in your 'clean room' testing (IE with newly created keychains and repos) but your keyfiles wind up causing issues anyway (which can occur if you used them through particular upgrades in the 2.0.X series, I believe), try "exporting and re-importing everything" (See: keybase/client#6923 (comment) )

@gthank = Is it possible to upgrade the gnupg version on the Linux falcon 4.4.0-43-Microsoft #1-Microsoft Wed Dec 31 ... x86_64 GNU/Linux system ? It seems like the best solution is to upgrade gpg there to something that plays nicer with the gpg on your other systems, and then export/re-import keys if needed.

Edit - I also made use of the SECRETS_GPG_COMMAND environment variable as described at http://git-secret.io/git-secret to choose between installed versions of GPG as needed (particularly on Ubuntu to choose between gpg and gpg2)

[gthank]

@joshrabinowitz Can you walk me through how you did this? If you did the git secret init and git secret tell steps on the centos box (and possibly even the ubuntu box?) I'd expect the git secret reveal step to work on any of them. I can't seem to get rid of the type 12 packets if I do the initialization on the mac. I can try to test what happens if I initialize on Ubuntu 16.04 and then git secret tell on my Mac a little later today.

EDIT: It's not really practical to upgrade the GnuPG version on the Windows box. We're using GnuPG straight out of apt. I looked into compiling from source to get a current version, and that requires also building from source for like 8 dependencies, none of which built cleanly straight out of the tarball. Eventually the dev who owns that box and I gave it up as a bad job, which is when I gave up and just did everything on the old Ubuntu server.

[joshrabinowitz]

@gthank when you say you're using 'gpg straight out of apt', you mean the apt included in windows10 WLS? If so, can the ubuntu subsytem be upgraded?

Also have you tried using gpg2 2.2.6 from OSX's 'brew'? That's what I tested with on the Mac.

Please let us know if you discover anything about the exact order of init/tell/hide/reveal on the different systems/gpg versions to either trigger or avoid the issue.

It would be best if we could provide instructions on how to fix this issue ( skipped packet of type 12 in keybox) rather than doing some hacky workaround. If possible.

[gthank]

@joshrabinowitz Yes, I mean the apt in WSL. As far as I'm aware, the subsystem is not upgradeable outside of Windows updates, but I don't have any Windows machines to realistically test it. I could ping my Windows dev, but he hadn't used the WSL stuff until I asked him to so we could use git-secret.

I have not checked the GnuPG from Homebrew; I can work try that out later today, probably.

Initial Steps

On my Mac

1. git secret init
2. About 4 git secret tell operations
3. A bunch of git secret add operations
4. git secret hide
5. Push to shared git server

On the other machines

1. Pull the changes from above
2. git secret reveal
3. files not revealed

Workaround Steps

On Ubuntu server

1. git secret init
2. About 4 git secret tell operations
3. A bunch of git secret add operations
4. git secret hide
5. Push to shared git server

On my Mac

1. Pull the changes from above
2. git secret reveal
3. files revealed

[joshrabinowitz]

@sobolevn what do you think we should do about this?

[sobolevn]

@joshrabinowitz I think we should not fix this issue from our side.

This comment says that reimporting works: keybase/client#6923 (comment)
So, a fix in the documentation should be fine, I guess.

Maybe we can even write some helper script to import/export these keys properly?

[joshrabinowitz]

@gthank I'm planning on adding some documentation about this.

Can you please try the export/import like the above keybase issue suggests and let us know if that works for you?

[joshrabinowitz]

Also, in my testing, I was able to replicate the gpg: skipped packet of type 12 in keybox error.

[gthank]

@joshrabinowitz I'm actually fairly booked up for the next couple of days. Can you give me a 50,000 ft overview of what import / export actually involves?

[joshrabinowitz]

@gthank I just ran through the export/import process:

• backup your gpg configs (cp -rp ~/.gnupg ~/.gnupg-bak)

• export the config (gnupg --export EXPORT.gpg)

• re-import the config (gnupg --import EXPORT.gpg)

If that doesn't might be necessary to do the export on one system (probably the one where everything works), copy the file to the other system, and do the import there.

[jcrben]

From my perspective, there's not an ideal solution to this from the git-secret side of things: upstream broke something they shouldn't have

Has anyone reported this upstream? Here's a query for kbx in their Phabricator:
https://dev.gnupg.org/maniphest/query/ZZUHMpykBqFQ/#R - also a query for error and keybox didn't turn up anything looks like as noted above a mailing list message was acknowledged by Werner https://lists.gnupg.org/pipermail/gnupg-devel/2017-May/032846.html

It sounds like they turned off jenkins (https://lists.gnupg.org/pipermail/gnupg-users/2018-January/059817.html) and they don't follow any sort of semantic versioning so future breakages (or annoyances, I know this isn't technically breaking) shouldn't be much of a surprise. I started suppporting https://www.patreon.com/neopg

According to https://lists.gnupg.org/pipermail/gnupg-users/2017-September/059096.html using GPGME is recommended for a (more) stable interface.

[sobolevn]

@jcrben thanks for gpgme.
That's a totally a thing to try.

[simbo1905]

I get why the external format of a key is stable (people exchange keys) yet the key ring format may not be forwards compatible (people upgrade, but dont downgrade, and it's unusual to be sharing key rings rather than keys).

How about we switch to storing keys in the public key format and don't commit the keyring that can "cache them" for repeated use:

1. git secret tell exports the key into a .gitsecret folder in the standard export format
2. git secret hide tries to encrypt using a keyring marked as .gitignore so as not to be committed:

• If there is isn't yet a keyring at the ignored location initialise it
• Speculatively try to encrypt with every identity. If there is a "key not found error" load the corresponding public key into the ignored keyring and retry

The advantages of this approach should be:

• if the user upgrades to some future gpg then that is still likely the able to load the keys in their local ignored keyring. The upgrade doesn't pollute any users ignored keyrings.
• If a user downgrades gpg and can no longer open the ignored keyring they simply delete it. We could provide a git secret clean command to do that. Meanwhile it is a problem for only for that one user.
• teams can all use different distributions of gpg and pgp as long as all versions understand the public key export format which is typically stable and new software will be tested against old key formats

That's just a rough sketch I may have missed something but fundermentally using a shared key ring rather than exported public keys is an "optimisation" over the approach above. Our current "optimisation" of a shared public key ring is causing teams who use different distributions or operating systems a compatibility challenge. So we should "downgrade" our approach to have logic that uses a local ignored keyring as a "cache".

[sobolevn]

I like the idea of moving away from managing keyrings by ourself.
Currently I can not even see any disadvantages in this approach.

[sobolevn]

@notjames as I understood @simbo1905's suggestion, we are not going to use the default keyring in $HOME. We are going to still use the same keyring location as we use now.

But, this keyring will be ignored by the git. This way we can recreate a keyring on every machine with just importing the public keys.

[notjames]

OK. I think I'm able to work through in my head how this will work. The part that I was not thinking about is the actual encrypted files which have been encrypted by specific keys as being the so-called "access list" I was on about.

That said, why use the public keyring in the repo at all? Why not just use each user's $HOME/.gnupg/pubring.kbx? Why have a different keyring somewhere else at all? What does that buy us?

[sobolevn]

Because we do not want to pollute user's main keyring with our keys.
That was the main point for me in the past.

[notjames]

An interesting thing to want to avoid since the keyring is "polluted" by public keys already anyway. My keyring has keys from me, co-workers, public keys from OS maintainers, etc. I don't and have never really cared about what's in there. I think it's good to be cognizant of good housekeeping, but I believe you cause potentially more issues by not keeping the keyring in a well-known backed up place. If someone were to lose the keyring for whatever reason, would it be difficult to recover the keys required to re-encrypt the files? I'm not so sure it's a good idea to have a keyring that isn't backed up somewhere. Currently, that is the repo. If that is removed, recovery could end up being a problem. Putting the keyring in the repo but not backed up is something people who have knowledge of how this system works can deal with, but to the lay-user, not-so-much.

[joshrabinowitz]

I think what @simbo1905 is suggesting is that .gitsecret/keys contain one or more files with text representations of the public keys of the users that can access the repo, and that these text file(s) would get checked in to git.

The file(s) of public keys in text representation would be portable across gpg versions. (You can see a public key in text representation with: gpg --export --armor email@id.)

The actual gpg keychains used on each given system (for each repo using git-secret) would be created, cached, and re-created from the public keys (and at least one of the needed private keys, which are probably in the user's keychain in ~/.gnupg) as needed.

The idea is that keychain files, which differ across gpg versions, would not be checked into the git repo.

In this way git-secret would avoid issues with different systems' versions of gpg reading and writing files in .gitsecret/keys differently, as these files would not get checked into the git repo or shared between systems.

[notjames]

@joshrabinowitz OK. I think I understand better by your description what @simbo1905's suggestion is. It all sounds OK, only maintaining the flat-file ascii-armored keys might become a pain in the butt...

[simbo1905]

At https://stackoverflow.com/a/22147722 it says:

Machine-Readable Output

GnuPG also has a colon-separated output format, which is easily parsable and has a stable format. The format is documented in GnuPG doc/DETAILS file. The option to receive this format is --with-colons.

$ gpg --with-colons a4ff2279.asc
gpg: WARNING: no command supplied. Trying to guess what you mean ...
pub:-:8192:1:4E1F799AA4FF2279:1356475387:::-:
uid:::::::::Jens Erat (born 1988-01-19 in Stuttgart, Germany):
uid:::::::::Jens Erat [email protected]:
uid:::::::::Jens Erat [email protected]:
uid:::::::::Jens Erat [email protected]:
uid:::::::::Jens Erat [email protected]:
uat:::::::::1 12921:
sub:-:4096:1:0F3ED8E6759A536E:1356517233:1482747633:::
sub:-:4096:1:2D6761A7CC85941A:1356517456:1482747856:::
sub:-:2048:1:9FF7E53ACB4BD3EE:1358985314:1674345314:::
sub:-:2048:1:5C88F5D83E2554DF:1358985467:1674345467:::
sub:-:4096:1:8E78E44DFB1B55E9:1395870592:1599164118:::
sub:-:4096:1:CC73B287A4388025:1395870720:1599164118:::
sub:-:4096:1:382D23D4C9773A5C:1416680427:1479752427:::
sub:-:4096:1:FF37A70EDCBB4926:1456322829:1582466829:::

Here are the docs of the format https://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=blob_plain;f=doc/DETAILS

Running it over my old key I see:

$ gpg --keyid-format long --with-colons simon.public.key 2>/dev/null
pub:u:1024:17:B1973C29239EC912:1403553506:::u:
uid:::::::::Simon Massey (simon at my full name dot org) [email protected]:
sub:u:2048:16:47DAEA9F3754DBAE:1403553506::::

The first line has B1973C29239EC912 which looks to be my public key ID. So refining my sketch a little (still an approximation):

• git secret tell will run export against the user's personal keyring into a temp file, then run --with-colons over it to get the metadata into a second temp file, then move both the key and the meta into the .gitsecret/keys folder renaming them to use the "public key IDs" as the file name (e.g. looking at the output above my key could give B1973C29239EC912.pub.key and B1973C29239EC912.colon.txt). These files get committed to git.
• git secret hide will run ls -1 .gitsecret/keys/*.pub.key to get the list of all "public key IDs" that should be in the ignored keyring. Then run gpg --list-keys against the ignored keyring getting the list of "public key IDs" currently cached in it. Calculate the missing keys and import them into the keyring. Then run as it does today.

There might be other commands that need to be altered, so that is just a high-level approach.

As it looks like we can use a documented machine-readable format that should be stable, it looks to me like we should be safe. A simple tool like awk can probably compute the list of keys to import for git secret hide.

[simbo1905]

For bonus marks I think we can make the new approach forwards compatible with the current approach. Let's say we version the new approach 0.3.0. We want it to be forward compatible with the current 0.2.x. It's likely that any team that stated on 0.2.x will have one new team member join who first installs 0.3.0:

1. The new version will look for .gitsecret/cache and if it doesn't exist tell the user "This repo was initialised by a previous version of git-secret, please run git secret migrate to upgrade. The migration will keep backwards compatibility with older versions."
2. migrate can export out all the keys from the old keyring into the new format described above. It can the create the cache folder, add it to .gitignore, and create the ignored personal keyring "cache" within that folder, and load in all the exported keys into it.
3. The new version of will now test for the existence of the old public shared keyring when running commands. If it finds one it wil run the old logic to keep it updated.

This would mean:

• Each user of an existing project that upgrades to 0.3.x will get a more stable experience
• Users still on 0.2.x will work as now. They can still hit the keyring compatibility issue. The first thing they are likely to do is upgrade to the latest 0.3.x which will solve their problem.
• Teams that all start out on 0.3.x will just execute a single file exists check on the old keyring location and skip the backwards compatibility logic.

We will have to maintain the old logic in the code base for a while. We can give a deprecated warning and at some future 0.4 (or 1.0!) drop that logic. I think backwards compatibility is worth the effort though as the new logic simply needs to runs the old logic and have a migrate step. We can keep an eye on how much maintenance the old logic needs and make a call to ditch it sooner rather than later if the burden is high.

So how do we make the decision to adopt the proposed approach? One idea is I could make an "RFC001.md" containing the described approach and send a PR. We can then do a code review of it and update it and use the usual "LGTM" approach. Once we are happy to merge to master its the new design. Then we can open a fresh ticket to implement it. That would avoid these long and heavy comments in the future!

[sobolevn]

+1 for this roadmap. Thanks @simbo1905
@joshrabinowitz what's your opinion?

[simbo1905]

here is a first draft of the first part of the design as a PR #207

[joshrabinowitz]

@simbo1905 @sobolevn I like the RFC, I think it's a big step forward. Would love to see 0.2.x released before this development starts though!

[simbo1905]

The ticket for the long-term fix is #208

[florabtw]

For anyone else arriving here, I created a Docker image that contains git-secret and gpg 2.2 to solve this problem for me. The repo (with usage documentation) is here:

https:/ncpierson/git-secret

[sobolevn]

@ncpierson awesome! Thanks!

[emacdona]

I see this is still open. I ran into it today. Initialized git secret on my mac, gpg version: 2.2.27, but can't reveal secrets on our CI/CD server: Ubuntu 16.04 (xenial); gpg version: 2.1.11.

Are there any plans to implement a fix? And is there any documentation for the export/import of keys? In the meantime, I may try this docker workaround.

Thanks!

[joshrabinowitz]

I think the fix is to use armor format keys instead of the binary ones (See #623).
Actually fixing the problem with binary keys is very hard.

[joshrabinowitz]

That's good input about documenting how to export and import keys, I'll add a ticket about that @emacdona

[jcrben]

See also #631

[joshrabinowitz]

See also #739 which appears to be a gnupg version interoperability issue too.

[joshrabinowitz]

the causes and implications of this and related issue are pretty well documented. Closing this