ci: Test opt-usrlocal-overlays end-to-end in Prow CI

[jlebon]

Since both Prow and CoreOS CI do a compose and run kola tests, we
can change one of them to cover some different options without losing
coverage on the default path.

Follow-up to #4728.

[jlebon]

OK weird, for some reason rpm-ostree-2024.2-2.fc39.x86_64.rpm doesn't have /usr/lib64/rpm-ostree/rpm-ostree-0-integration-opt-usrlocal-compat.conf. Will investigate.

[HuijingHei]

Seems /usr/lib64/rpm-ostree/rpm-ostree-0-integration-opt-usrlocal-compat.conf is added by 3dc18c, which is not included in any release yet.

But during rsync -rlv /cosa/component-install/ overrides/rootfs/, can find usr/lib64/rpm-ostree/rpm-ostree-0-integration-opt-usrlocal-compat.conf.

[jlebon]

Seems /usr/lib64/rpm-ostree/rpm-ostree-0-integration-opt-usrlocal-compat.conf is added by 3dc18c, which is not included in any release yet.

But during rsync -rlv /cosa/component-install/ overrides/rootfs/, can find usr/lib64/rpm-ostree/rpm-ostree-0-integration-opt-usrlocal-compat.conf.

Yes good catch, thanks. I misremembered and thought I had added rpm-ostree-0-integration-opt-usrlocal-compat.conf as part of #4763.

[jlebon]

I just moved over Prow to build the RPM too in #4819. Once that merges, I'll rebase this.

[jlebon]

/retest-required

[jlebon]

Prow infra not having a good time apparently.

[jlebon]

/retest

[jlebon]

OK right, Prow CI needs coreos/fedora-coreos-config#2838 to work.

[jlebon]

/retest

[jlebon]

Ahh OK, the state overlay stuff is breaking with apply-live because the transient /usr mount shadows the overlay mounts. I think the simplest is probably to have ostree_sysroot_deployment_unlock() to know to recreate the overlay mounts afterwards? The state overlays will show up twice in the mount graph but meh, you can only unlock once, so twice is the worst case. (Hmm, if/once mount-beneath becomes available, possibly we could have the /usr overlay mount use that instead.)

[cgwalters]

Yeah, there's similar clash between usroverlay and systemd-sysext I think. A yet further wrinkle here is that in the composefs case there is no ostree-generated /usr bind mount - while usroverlay works what we probably really want is an ostree enable-transient verb that just flips the / composefs to gain a transient writable backing overlay (I think that's possible, if not we can just make a new composefs-overlayfs mount and swap).

[jlebon]

/retest

[HuijingHei]

• Get error:

qemu0 kola-runext-misc.sh[2242]: Feb 22 21:33:42 localhost systemd-tmpfiles[1153]: /sysroot/usr/lib/tmpfiles.d/rpm-ostree-autovar.conf:76: Duplicate line for path "/sysroot/var/opt", ignoring.
qemu0 kola-runext-misc.sh[2242]: Feb 22 21:33:54 localhost systemd-tmpfiles[1496]: /usr/lib/tmpfiles.d/rpm-ostree-autovar.conf:76: Duplicate line for path "/var/opt", ignoring.
error: Should not get logs (Duplicate line)

As we have another different definition for /var/opt which is from package filesystem, this is harmless, I think.

[root@cosa-devsh tmpfiles.d]# grep /var/opt * -rn
rpm-ostree-0-integration-opt-usrlocal-compat.conf:5:L /var/opt - - - - ../usr/lib/opt
rpm-ostree-autovar.conf:76:d /var/opt 0755 root root - -   (--> comes from /usr/lib/rpm-ostree/tmpfiles.d/filesystem.conf)

• Another failed log is localhost systemd-tmpfiles[1642]: Failed to create directory or subvolume "/usr/local/man": Read-only file system, but other directories like /usr/local/bin are all created.

[jlebon]

OK, I think I got it all now! Tests which use apply-live are just skipped for now. Commit "compose: Inject our static tmpfiles.d dropins earlier" is the only non-trivial one; that one probably deserves more attention.

[jlebon]

/retest

[cgwalters]

Looks generally sane to me. I still have some concerns about the state overlay approach, but that's not a reason to block it.

[jlebon]

I still have some concerns about the state overlay approach, but that's not a reason to block it.

Here's how I see it: the transient root approach and state overlays approach both solve the /opt issue, but within different models of how the OS should work (the container model and the immutable model). The immutable model is still quite desirable (and I would argue should be the default) and solving the /opt issue there is still valuable.

[cgwalters]

This PR is probably not the best place to debate but...I wouldn't call stateoverlays "immutable" - they're very much about mutable persistent state.

[jlebon]

This PR is probably not the best place to debate but...I wouldn't call stateoverlays "immutable" - they're very much about mutable persistent state.

I'm not saying state overlays are immutable. Only that it solves the /opt issue within an otherwise immutable OS. It adds the minimum amount of mutability where needed to achieve that.