override replace on the systemd rpm does not replace /usr/bin/systemctl

[prydom]

Describe the bug

When using rpm-ostree override replace to upgrade packages in a container overlay, upgrading systemd does not replace the /usr/bin/systemctl binary.

Reproduction steps

1. podman run -it -v ~/rpms:/tmp/rpms:ro,z "$BASE_IMAGE:$BASE_TAG" /bin/bash --login
2. rpm-ostree override replace /tmp/rpms/systemd-255.5-1.fc41.x86_64.rpm (upgrading from systemd-255.4)
3. rpm -V --nomtime systemd

Expected behavior

/usr/bin/systemctl should match the version of the replaced RPM. The RPM should verify as expected below.

# rpm -V --nomtime systemd
.M.......  c /etc/machine-id
missing     /var/lib/systemd
missing     /var/lib/systemd/catalog

Actual behavior

/usr/bin/systemctl is the version from the base image. If libsystemd was also upgraded then /usr/bin/systemctl will fail to start due to missing shared libraries.

# rpm -V --nomtime systemd
.M.......  c /etc/machine-id
S.5......    /usr/bin/systemctl
missing     /var/lib/systemd
missing     /var/lib/systemd/catalog

System details

# rpm-ostree --version
rpm-ostree:
 Version: '2024.5'
 Git: f9c588bf86cc2ff00034bd7ea846392351aa5e47
 Features:
  - rust
  - compose
  - container
  - fedora-integration
# cat /etc/os-release
NAME="Fedora Linux"
VERSION="Rawhide.20240422.n.0 (Kinoite Prerelease)"
ID=fedora
VERSION_ID=41
VERSION_CODENAME=""
PLATFORM_ID="platform:f41"
PRETTY_NAME="Fedora Linux Rawhide.20240422.n.0 (Kinoite Prerelease)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:41"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://kinoite.fedoraproject.org"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-kinoite/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://pagure.io/fedora-kde/SIG/issues"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=rawhide
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=rawhide
SUPPORT_END=2025-05-13
VARIANT="Kinoite"
VARIANT_ID=kinoite
OSTREE_VERSION='Rawhide.20240422.n.0'

Additional information

This can be worked around by rebuilding the base image from a treefile. My base image uses Fedora Rawhide (quay.io/fedora/fedora-kinoite). In order to reduce the number of layers changed from day-to-day I use override replace in a container overlay to apply package updates using this script and Containerfile.

I rebase the entire container at least once a week to reduce but not eliminate hysteresis.

cliwrap is enabled.

[root@a7f254fd6474 /]# ls -l /usr/libexec/rpm-ostree/wrapped/
total 188
-rwxr-xr-x. 2 root root 95282 Dec 31  1969 dracut
-rwxr-xr-x. 2 root root 66000 Dec 31  1969 kernel-install
-rwxr-xr-x. 2 root root 20624 Dec 31  1969 rpm

[prydom]

This seems likely to be caused by the wrapping of systemctl and the various user modification tools at

rpm-ostree/rust/src/core.rs

Lines 143 to 151 in e0cf3f7

	impl FilesystemScriptPrep {
	/// Filesystem paths that we rename out of the way if present
	const OPTIONAL_PATHS: &'static [&'static str] = &[SSS_CACHE_PATH];
	const REPLACE_OPTIONAL_PATHS: &'static [(&'static str, &'static [u8])] = &[
	(GROUPADD_PATH, GROUPADD_WRAPPER),
	(SYSTEMCTL_PATH, SYSTEMCTL_WRAPPER),
	(USERADD_PATH, USERADD_WRAPPER),
	(USERMOD_PATH, USERMOD_WRAPPER),
	];

and so this probably also impacts rpm-ostree override replace of the shadow-utils package.

[prydom]

@cgwalters, while I have your attention 😄, is there a reasonable workaround for this?

I'd love to use dnf5 install or dnf5 upgrade instead of rpm-ostree override replace but I still use rpm-ostree for installation in container layers because it handles missing /usr/lib/tmpfiles.d entries and package script created users/groups for me.

[cgwalters]

We are backing away from cliwrap.

As far as dnf5...in general, please do ping those folks on their project with feature requests! (I talk to them too, sometimes we have join meetings but still...start there. Maybe we can add a label or something to issues there to denote they apply to bootc/rpm-ostree)

[prydom]

I will note that I don't actually think this bug has to do with cliwrap as rpm-ostree doesn't seem to replace systemctl all the time, just when in the rpm installation/scripts context.

Note that I do use cliwrap to support the kernel replacement kernel-install use case - I don't use the rawhide kernel, just the userland and so I need to replace it. Although, I do have a hacky shell script alternative that I wrote before I realized that I could enable cliwrap to do the same job.

Duly noted about the dnf5 request though. I have some python scripts that use their library but having the features be upstream would be preferred.