Ubuntu Revisiting Its Initramfs Compression Approach

About a year ago Ubuntu changed its default compression level for its initramfs handling down to Zstd level one to deal with slow initramfs creation times on low-end systems / development boards. But since then that has resulted in larger initramfs sizes and yielding other bugs like more quickly filling up the /boot partition on Ubuntu systems. Thus the developers have gone back to the drawing board and are trying to figure out a path forward for better initramfs handling that works well for low-end single board computers while also maximizing space savings and working out well for all Ubuntu use-cases.

Ubuntu/Debian developer Benjamin Drung has been working to find the right balance of compression options for Ubuntu's initramfs to be fast and yield good compression. His original proposal laid out last week on the ubuntu-devel list includes:

"I propose following approach to address the drawback: Create cpio archives (compressed with xz -9) for the kernel modules and firmware files when building the kernel/firmware Debian package. Then ship those cpio archives in the package (or in a separate binary package). Then the CPU load it put on the builders. The cpio archives would contain the modules for MODULES=most.

mkinitramfs will then look for those cpio archives and uses those in case they are present. Such a initramfs would look like this:

* AMD/Intel microcode cpio archive (on amd64)
* main cpio archive compressed with zstd -1
* kernel modules from the Debian package compressed with xz -9
* firmware files from the Debian package compressed with xz -9

After working on initramfs-tools as part my day job, my fingers were itching and I had to create a quick and dirty draft in my free night time. You can find the result of the last two hours in [1]. This draft has a mkinitramfs-kernel script that creates a cpio archive containing the kernel modules and firmware (that needs to be split later on).

The lunar test result on my AMD Ryzen 7 5700G look promising: Building 6.2.0-24-generic-modules-most.cpio.xz takes around 90 seconds and is 54.9 MiB in size. Creating the initramfs speeds up from around 8.7 seconds to 3.5 seconds (saves 60 %). The size reduces from 133.1 MiB to 80.7 MiB (saves 39.4 %). So the boot needs 52.4 MiB less, but /lib/modules need 54.9 MiB for the cpio archive.

The drawback is that building the kernel would take longer, the package takes more space on the archive and mirrors, and downloading them could take longer on slow connections."

Canonical engineer Dimitri John Ledkov meanwhile expressed another possible approach:

"Creating xz -9 compressed archive of all modules, still pays the penalty to decompress most of them, and then not modprobe them. I was hoping to achieve a similar in spirit approach, but didn't quite have the time to implement is:

1) change linux-modules and linux-firmware to ship .ko.zst firmware.bin.zst compressed with zstd -19 at .deb build time 2) this saves install size of the packages, with only slightly increased download size
3) modify initramfs-tools to include compressed files into a separate initrd, which is not compressed (i.e. exclude .zst files from the default main compressed cpio archive, and append them in the second main cpio archive that is uncompressed)
4) this should achieve quick initrd creation, which will be smaller in size that current status, and will boot faster as it will only decompress modules/firmware it actually needs at boot

For experimentation locally, you can recompress .ko with zstd in place in /lib/modules/; and rerun depmod. To then test initramfs-tools changes that skip over .zst compressed files and add them as is in an uncompressed appended cpio."

That path is currently being explored. Nothing is set in stone yet though but in any event it looks like something will happen for changing the initramfs compression handling on Ubuntu to provide a better balance between compression size / storage requirements, compression times, and decompression requirements to fit the wide range of hardware where Ubuntu sees use. Likely any change will happen for the Ubuntu 23.10 cycle to allow sufficient time for testing ahead of next year's Ubuntu 24.04 LTS release.