build-id conflict during RPM install

[philrz]

A community user reported on Slack:

trying to install brim 0.24.001 on Centos 8 and get the following conflict:

file /usr/lib/.build-id/1d/a3a1d77c7109ce6444919f4a15e7e6c63d02fa from install of brim-0.24.0-1.x86_64 conflicts with file from package Volatility3-1.0.1-2.el8.x86_64

$ ls -l /usr/lib/.build-id/1d/a3a1d77c7109ce6444919f4a15e7e6c63d02fa
lrwxrwxrwx. 1 root root 24 Mar 10 10:08 /usr/lib/.build-id/1d/a3a1d77c7109ce6444919f4a15e7e6c63d02fa -> ../../../../usr/bin/vol3

I'm not sure what the /usr/lib/.build-id/ path is for, but is that really a hash collision ?

This failure was unfamiliar, so I dug into it on a scratch CentOS 8 VM. It does seem like a legitimate collision, unfortunately. I started with having only Brim installed and I could see what that symlink is pointing at:

$ ls -l /usr/lib/.build-id/1d/a3a1d77c7109ce6444919f4a15e7e6c63d02fa
lrwxrwxrwx. 1 root root 73 Feb  8 18:43 /usr/lib/.build-id/1d/a3a1d77c7109ce6444919f4a15e7e6c63d02fa -> ../../../../usr/lib/brim/resources/app/zdeps/suricata/bin/suricata-update

Then I uninstalled Brim and installed Volatility3 and I can see that's pointing at a different file:

$ ls -l /usr/lib/.build-id/1d/a3a1d77c7109ce6444919f4a15e7e6c63d02fa
lrwxrwxrwx. 1 root root 24 Mar 10 10:08 /usr/lib/.build-id/1d/a3a1d77c7109ce6444919f4a15e7e6c63d02fa -> ../../../../usr/bin/vol3

The files are definitely different though. Having copied them out of those locations:

$ ls -l suricata-update vol3 
-rwxr-xr-x. 1 phil phil 5955624 Apr 26 21:12 suricata-update
-rwxr-xr-x. 1 phil phil 9761920 Apr 26 21:12 vol3

Having just learned about this for the first time, I don't claim a thorough understanding of what these "build-id" paths are for either, but quick web searches seem to imply it involves "unique identification of binaries" (https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/developer_guide/compiling-build-id). Using the commands they show there that generate the hashes, indeed, it seems to be true:

$ readelf -n suricata-update 
Displaying notes found in: .note.ABI-tag
  Owner                 Data size	Description
  GNU                  0x00000010	NT_GNU_ABI_TAG (ABI version tag)
    OS: Linux, ABI: 3.2.0
Displaying notes found in: .note.gnu.build-id
  Owner                 Data size	Description
  GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)
    Build ID: 1da3a1d77c7109ce6444919f4a15e7e6c63d02fa

$ readelf -n vol3 
Displaying notes found in: .note.ABI-tag
  Owner                 Data size	Description
  GNU                  0x00000010	NT_GNU_ABI_TAG (ABI version tag)
    OS: Linux, ABI: 3.2.0
Displaying notes found in: .note.gnu.build-id
  Owner                 Data size	Description
  GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)
    Build ID: 1da3a1d77c7109ce6444919f4a15e7e6c63d02fa

I think that leads me to a theory, though. This suricata-update binary happens to have been generated by using pyinstaller (https://www.pyinstaller.org/) to "freeze" the Python code in which Suricata Update is written, since that allows us to avoid having Python installed and with the correct dependencies on all users systems. And I see that Volatility also claims to be a Python project, so perhaps it's similarly packaged and that trips up the hashing somehow. Perhaps the hashing doesn't "see" the Python code parts that make each overall file different, so it comes away with a sense that they're equivalent? That's just a theory though.

I found a thread at https://stackoverflow.com/questions/21246680/disable-yum-transaction-check-for-file-conflict where people were debating whether it's safe to let these stomp on each other to resolve conflicts. I can't say with certainty what the correct answer is, but FWIW, since my CentOS 8 VM is just a scratch one, I did try sudo rpm -i --replacefiles brim_x86_64-v0.24.0.rpm while Volatility was already installed and it did, indeed, install without complaint and seemed to launch ok. I don't know Volatility well enough to check its health, however. The user that reported the problem did use this workaround and claimed that both Brim and Volatility seemed to be working ok for them.

I'm not sure what we might do to better avoid this class of problem in the future. A few that come to mind:

1. Perhaps there's a way to ensure our binary is seen as unique such that it generates a unique hash
2. Perhaps it would be appropriate/possible to package the RPM in a way that prevents these links from being created
3. Perhaps one day we'll opt for one of the packaging schemes described in Distribution-agnostic packaging (Flatpak, AppImage, Snap) #685 that I assume do not have this problem

For now I'll hold this issue open in the event other users stumble onto it. If anyone does and reads this, please add a comment with details of your experience.

[zmiimz]

I have the same case during rpm package installation

sudo rpm -ihv --nodeps zui-1.4.1.x86_64.rpm
Verifying... ################################# [100%]
Preparing... ################################# [100%]
file /usr/lib/.build-id/51/74642779eb830026a665bff7d1869207a574ac from install of zui-1.4.1-1.x86_64 conflicts with file from package xmind-vana-23.11.4336-202311170221.x86_64
file /usr/lib/.build-id/7b/bde14ca6192338b4c5d44af06e90ca06276940 from install of zui-1.4.1-1.x86_64 conflicts with file from package xmind-vana-23.11.4336-202311170221.x86_64
file /usr/lib/.build-id/8b/060bea4cd4260d05f778be1856691d20377fcb from install of zui-1.4.1-1.x86_64 conflicts with file from package xmind-vana-23.11.4336-202311170221.x86_64
file /usr/lib/.build-id/90/9d0b5007ba7bf20d9e4205235b3fe5a9f3309e from install of zui-1.4.1-1.x86_64 conflicts with file from package xmind-vana-23.11.4336-202311170221.x86_64

[philrz]

Thanks for letting us know @zmiimz. We unfortunately have not been able to fix this. Please do let us know if you were able to workaround the problem with the --replacefiles approach described above or any other ways.

[philrz]

Per #3084 (comment), I've noticed that this can also occur when attempting to install a Zui RPM when a Zui Insiders release is already installed.