Fix two external issues breaking cfs on latest f44+ by cgwalters · Pull Request #2195 · bootc-dev/bootc

cgwalters · 2026-05-11T12:22:13Z

Yes, we really do need to add at least our basic TMT tests as gating on Fedora packages.

dracut 109 (shipped in Fedora 45, and now backported to Fedora 44 via FEDORA-2026-5e95ea992d) removed `systemdsystemconfdir` from 01-dist.conf, leaving the variable empty in the dracut build environment. Our module-setup.sh used that variable for the `initrd-root-fs.target.wants/` directory, so the symlink was written to the root of the initramfs rather than under /usr/lib/systemd/system/. systemd does not recognize root-level .wants directories, so bootc-root-setup.service was silently never activated, causing the system to drop to emergency mode on composefs+UKI boots. Switch to `${systemdsystemunitdir}` (/usr/lib/systemd/system), which dracut reliably sets on all versions. This matches how the ostree dracut module has always handled this. The fix is unconditionally correct for all dracut versions. Assisted-by: OpenCode (Claude Sonnet 4.6) Signed-off-by: Colin Walters <walters@verbum.org>

Linux kernel 7.0 and 7.1 introduced a requirement that all files in a composefs image carry fs-verity digests. bootc only applies verity to content objects, not every file, so erofs mounts fail with 'has no fs-verity digest' on these kernels. This is a kernel regression tracked upstream; 7.2+ is expected to restore the prior behavior. In provision-fetch.sh, detect kernel 7.0.x/7.1.x in the base image and downgrade to 6.19.10-300.fc44 from Koji before the image build runs. The downgrade uses --noscripts to prevent the kernel-core posttrans scriptlet from invoking kernel-install (which fails and pollutes /boot inside the container), then runs depmod manually and removes leftover module directories from the replaced kernel. This workaround should be removed once base images ship kernel 7.2+. Assisted-by: OpenCode (Claude Sonnet 4.6) Signed-off-by: Colin Walters <walters@verbum.org>

gemini-code-assist

Code Review

This pull request updates the initramfs module setup to use the systemd unit directory for target dependencies and introduces a temporary kernel downgrade in the provisioning script to address a composefs issue in kernel versions 7.0 and 7.1. Review feedback highlights potential issues in the downgrade logic, specifically regarding multi-version kernel detection and the need to explicitly remove the problematic kernel packages to ensure the downgraded version is correctly prioritized.

gemini-code-assist · 2026-05-11T12:24:16Z

+        # which calls rpm-ostree, and that fails inside a container build.
+        # We manually run depmod afterward since it's the only useful
+        # scriptlet the kernel packages would otherwise execute.
+        dnf -y install --allowerasing --setopt=tsflags=noscripts "${kernel_td}"/*.rpm


Using dnf install on an older kernel version will install it alongside the existing 7.x kernel rather than replacing it. Since the 7.x kernel has a higher version number, it will likely remain the default boot option. Deleting the module directory manually (lines 167-172) will leave the 7.x kernel in a broken state (missing modules) while still being the default. It is better to explicitly remove the 7.x kernel packages to ensure the 6.x kernel is preferred and the RPM database remains consistent.

Suggested change

dnf -y install --allowerasing --setopt=tsflags=noscripts "${kernel_td}"/*.rpm

dnf -y install --allowerasing --setopt=tsflags=noscripts "${kernel_td}"/*.rpm

dnf -y remove --setopt=tsflags=noscripts "kernel*-7.[01]*"

Good thought Gemini, but we actually set installonlyn in our base image to turn off that dnf behavior.

jeckersb · 2026-05-12T18:38:58Z

Hm test-upgrade (fedora-43, ostree) is consistently failing with:

$ bcvk libvirt run --name bootc-tmt-8uu8isbc-tmt-plans-integration-plan-01-readonly --detach --label=bootc.test=1 --bind-storage-ro localhost/bootc-upgrade-source
Filtered from 31 to 1 plan(s) based on arguments: ["readonly"]
Found 1 test plan(s): ["/tmt/plans/integration/plan-01-readonly"]

========================================
Running plan: /tmt/plans/integration/plan-01-readonly
VM name: bootc-tmt-8uu8isbc-tmt-plans-integration-plan-01-readonly
========================================

Creating libvirt domain 'bootc-tmt-8uu8isbc-tmt-plans-integration-plan-01-readonly' (install source container image: localhost/bootc-upgrade-source)
 INFO Creating default storage pool at "/home/runner/.local/share/libvirt/images"
 INFO Default storage pool created successfully
 INFO Creating base disk: "/home/runner/.local/share/libvirt/images/bootc-base-5b66923bd34d6412.qcow2"
Error: OCI runtime error: crun: the container `76afad9f8a589b6860a9d9143add843ad779c00610ab4714ea431f88430c03b8` is not running
Error: no container with name or ID "76afad9f8a589b6860a9d9143add843ad779c00610ab4714ea431f88430c03b8" found: no such container
(Container produced no output)
Error: 
   0: Failed to find or create base disk
   1: Failed to install bootc to base disk: "/home/runner/.local/share/libvirt/images/bootc-base-5b66923bd34d6412.TmxGIo.tmp.qcow2"
   2: Monitor process exited unexpectedly: ExitStatus(unix_wait_status(65280))

Location:
   crates/kit/src/run_ephemeral_ssh.rs:246

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
   1: __libc_start_main<unknown>
      at <unknown source file>:<unknown line>

Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.
Failed to launch VM for plan /tmt/plans/integration/plan-01-readonly: Launching VM with bcvk: command exited with non-zero code `bcvk libvirt run --name bootc-tmt-8uu8isbc-tmt-plans-integration-plan-01-readonly --detach --label=bootc.test=1 --bind-storage-ro localhost/bootc-upgrade-source`: 1

error: Running TMT tests: Some test plans failed
========================================
Test Summary
========================================
/tmt/plans/integration/plan-01-readonly: FAILED
========================================

Not immediately sure what that's all about, I can run just test-upgrade successfully locally against f43/ostree 🤷

jeckersb · 2026-05-12T20:20:17Z

Trying to bump to the latest bcvk release to see if that magically fixes it.

jeckersb · 2026-05-12T21:18:01Z

Well same issue with newer bcvk, although it occurs to me that we should probably be testing fedora-44 now in the upgrade matrix so let me try switching that out and see if it makes any difference as well (probably we should test both stable fedora releases but if we can get by with changing to fedora-44 for the moment that's progress).

Really we should probably test both of these while they're stable Fedora versions, but currently CI is blocked on the f43 upgrade job being broken, and if switching to f44 unblocks things that's progress. Signed-off-by: John Eckersberg <jeckersb@redhat.com>

For fedora 44 we install kernel v6.19 ourselves and not run the postprocess scripts that would've generated the initramfs. This causes bcvk to fail with `No kernel found` So we explicitly regenerate the initramfs in Dockerfile.upgrade Signed-off-by: Pragyan Poudyal <pragyanpoudyal41999@gmail.com>

Johan-Liebert1 · 2026-05-13T06:41:24Z

Test Upgrade passes now

cgwalters · 2026-05-13T13:09:51Z

Ugh I'm sorry I created a mess here because I somehow pushed some early agent-generated garbage instead of a later cleaned up iteration. (It's complicated, I'm traveling w/o my usual devenv local, plus various other compounding things).

The root brokenness here is the --noscripts - and that is because we happened to uncover a new rpm-ostree bug where our initramfs handling breaks with /tmp being a separate filesystem. Will do followups here.

cgwalters added 2 commits May 11, 2026 08:19

cgwalters changed the title ~~Fix two external issues breaking cfs on f44+~~ Fix two external issues breaking cfs on latest f44+ May 11, 2026

cgwalters added the ci/merge Run full CI suite (all OSes) — equivalent to merge queue label May 11, 2026

bootc-bot Bot requested a review from jeckersb May 11, 2026 12:22

gemini-code-assist Bot reviewed May 11, 2026

View reviewed changes

cgwalters enabled auto-merge May 11, 2026 15:15

cgwalters mentioned this pull request May 11, 2026

Sync common files from infra repository #2193

Merged

jeckersb previously approved these changes May 11, 2026

View reviewed changes

jeckersb dismissed their stale review via bae04c0 May 12, 2026 20:19

jeckersb force-pushed the fix-composefs-boot-f45 branch from bae04c0 to bb8e7dd Compare May 12, 2026 21:22

jmarrero mentioned this pull request May 13, 2026

Test if this fixes CI... #2197

Closed

Johan-Liebert1 approved these changes May 13, 2026

View reviewed changes

cgwalters added this pull request to the merge queue May 13, 2026

Merged via the queue into bootc-dev:main with commit 983c09f May 13, 2026
54 of 62 checks passed

cgwalters deleted the fix-composefs-boot-f45 branch May 13, 2026 12:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix two external issues breaking cfs on latest f44+#2195

Fix two external issues breaking cfs on latest f44+#2195
cgwalters merged 4 commits into
bootc-dev:mainfrom
cgwalters:fix-composefs-boot-f45

cgwalters commented May 11, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

gemini-code-assist Bot May 11, 2026

Uh oh!

cgwalters May 11, 2026

Uh oh!

jeckersb commented May 12, 2026

Uh oh!

jeckersb commented May 12, 2026

Uh oh!

jeckersb commented May 12, 2026

Uh oh!

Johan-Liebert1 commented May 13, 2026

Uh oh!

Uh oh!

cgwalters commented May 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	dnf -y install --allowerasing --setopt=tsflags=noscripts "${kernel_td}"/*.rpm
	dnf -y install --allowerasing --setopt=tsflags=noscripts "${kernel_td}"/*.rpm
	dnf -y remove --setopt=tsflags=noscripts "kernel-7.[01]"

Conversation

cgwalters commented May 11, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

gemini-code-assist Bot May 11, 2026

Choose a reason for hiding this comment

Uh oh!

cgwalters May 11, 2026

Choose a reason for hiding this comment

Uh oh!

jeckersb commented May 12, 2026

Uh oh!

jeckersb commented May 12, 2026

Uh oh!

jeckersb commented May 12, 2026

Uh oh!

Johan-Liebert1 commented May 13, 2026

Uh oh!

Uh oh!

cgwalters commented May 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants