Linux EFI Zboot Abandoning "Compression Library Museum", Focusing On Gzip & Zstd

[Anux]

Normal people have huge kernels. The size is at least 200 MB (compressed) with firmware and initramfs. Maybe even as much as 500 MB in the coming years.

That sounds much to much. A random Mint install has (zstd compressed) 115 MB. I think uncompressed is 200 MB. With modules set to "dep", compressed size goes down to 90 MB.

My Arch has 50 MB (with zstd --long -6) and 70 MB for the fallback. Read speed of my SSD is ~3000 MB/s decompression time somewhere below 0.1 s. So the whole read and decompress step happens in 0.1 s and I honestly couldn't feel any difference to uncompressed. I just compress for the space savings and use a setting that doesn't impact initrd generation time.

[skeevy420]

That is true but less relevant to compressed kernel images.

I agree. It's rather moot in the context of kernel images since they're not compressed inline and, instead, compressed with something like Dracut or mkinitcpio. Moreover, inline compression is a file system thing. With the right setup, an EFI loader with kernels on a different file system, we'll still be able to use all those compressors ... with OpenZFS

Eh, on my Steam Deck it is fine to use btrfs zstd compression for games from microSD card, it gives a measurable reduction in loading times (plus I can store more games on a card, yay!).

That's to be expected on that kind of storage medium. For example, on a single 7200 RPM HDD I was able to set BTRFS and OpenZFS to use to Zstd 7 before I'd notice any bottleneck from Zstd due to how slow HDDs inherently are. The default value is 3. I'm not really sure what a good value is for a microSD.

That assumes a person always cares about write speed. When it comes to my mostly read-only data like games and places like /usr I use Zstd 19 with OpenZFS. My Steam download write speed is abysmal . Prefixes use LZ4 so games aren't hit with any Zstd 19 quicksave lag . I love OpenZFS's per dataset compression. Damn that's handy.

However, if you want to not bottleneck I/O from NVMe then tough luck, no CPU and compression algorithm will achieve that without hardware acceleration. Sony even spent considerable silicon area in PS5's storage chip in order to widen that bottleneck.

Without hardware acceleration we're basically left with LZ4 which limits us to OpenZFS or DM-VDO, and even LZ4 can be a bottleneck on the fastest of drives even when paired with the fastest of processors. Ironically, those really fast drives show that LZ4 can have the same limitations that Zstd has with BTRFS. LZ4 also has negative, fast, levels and drives are now fast enough that LZ4's default settings aren't necessarily good enough like they used to be and neither LZ4 nor Zstd have access to those levels at the kernel level...I'm ignoring that OpenZFS does have access to the Zstd fast levels for a second there...Like, terrific, we have to go out of tree for better Zstd support ...

[skeevy420]

I don't get it: NVMe is supposed to be fast, and attached directly to CPU lanes. Why would compression be a benefit, or rather how and why is the additional overhead of compression/decompression somehow sometimes faster than not having that overhead? A 1GB file should do whatever it needs to do at the speed of the hardware without involving another subsystem and CPU, and if disk I/O somehow can't keep up, it sounds like I need to RAID0 some more NVMes

I get embedded needing small stuff, but I'm talking Btrfs and mainstream stuff on desktop distros.

Well, if your drive runs a 3000Mb/s but LZ4 runs its CODEC operations at 6000Mb/s with your CPU then compressing your files with LZ4 means they'll be read and stored faster than the drive is normally capable. You'll be able to read a 1GB file compressed down to 768Mb faster than you can read a 1GB uncompressed file. That's the "free performance" I was talking about before.

Now, if you pair a really, really fast drive with a really, really slow processor, an NVMe with a Q6600, that'll be interesting

[Espionage724]

Compression is very important because harddisk place costs a lot of money !

I have everything imaginable installed and am using 18% of a 1TB NVMe. I kind of like stuff uncompressed to try to justify the space

[Daktyl198]

If you mean LZMA1 implementation, that has not been ignored but closely looked at and found wanting (xz too by the way).
Other LZMA implementations exist, e.g. inside 7-zip and lzip, but they are not materially different from xz when it comes to compression ratio and speed.

LZMA 1 and 2 are almost identical, except that LZMA2 allows for multi-process encoding/decoding. I'd love a different source on "closely looked at and found wanting" given that XZ has dominated the Linux space for the last 10 years. Additionally, the link you provided specifically had gripes about the XZ container format rather than LZMA, which makes sense given that it was written by the author of another LZMA implementation lol.

Zstd is great as a jack of all trades, I just think the majority of regular people would rather have a smaller file size and spend 3 seconds decompressing it rather than have a larger file size and spend 2 seconds decompressing it in a normal user workflow. Zstd and gzip seem more like they're meant to be used as tools by other software where speed is important.

[Phoronos]

I have everything imaginable installed and am using 18% of a 1TB NVMe. I kind of like stuff uncompressed to try to justify the space

Why so many people are using a lot of time and energy trying to find solutions for compressing the kernel image then ?

[Anux]

Why so many people are using a lot of time and energy trying to find solutions for compressing the kernel image then ?

Limited space on the boot/efi partition.

[Espionage724]

Limited space on the boot/efi partition.

Just increase the size

(or complain to your distro maintainers using small defaults )

[billyswong]

LZMA 1 and 2 are almost identical, except that LZMA2 allows for multi-process encoding/decoding. I'd love a different source on "closely looked at and found wanting" given that XZ has dominated the Linux space for the last 10 years. Additionally, the link you provided specifically had gripes about the XZ container format rather than LZMA, which makes sense given that it was written by the author of another LZMA implementation lol.

Conflict of interest indeed, but his opinion may still well be valid.

Personally I expect a compression file format to either be a complete archive format like .zip or .7z, or just focus on 1 compression algorithm if it requires the accompany of .tar. The .xz format being designed to be "extensible" to other compression algorithm in the future smell wrong to me. If one want a new algorithm in the future, it should have been done in the form of .tar.xz2 or whatever.

Zstd is great as a jack of all trades, I just think the majority of regular people would rather have a smaller file size and spend 3 seconds decompressing it rather than have a larger file size and spend 2 seconds decompressing it in a normal user workflow. Zstd and gzip seem more like they're meant to be used as tools by other software where speed is important.

I agree with the kernel boot image compression. For most people, slightly longer boot time is okay for disk space saving, especially if it is just a 1 second difference. The EFI partition or the /boot partition being filled up is a bigger concern.

[chithanh]

That tested on essentially just one dataset comprised of text.

If we're talking about kernel images, why don't you consider compression ratios on compiled code?

You are right that the comparison is less valid when comparing different classes of algorithms like LZ77 vs. BWT, CM, or PPM. But from experience if one LZMA implementation beats another in text compression it does also better in binary compression.

LZMA 1 and 2 are almost identical, except that LZMA2 allows for multi-process encoding/decoding. I'd love a different source on "closely looked at and found wanting" given that XZ has dominated the Linux space for the last 10 years. Additionally, the link you provided specifically had gripes about the XZ container format rather than LZMA, which makes sense given that it was written by the author of another LZMA implementation lol.

Yes, I specifically meant the LZMA1/LZMA2/XZ implementations including the container format. LZMA class algorithms themselves are fine, though no longer among the best. The arguments criticizing the XZ container format speak for themselves I think.

XZ has dominated the Linux space for the last 10 years.

There are other factors at play besides the technical merit. Most importantly, network effects made it entrenched and tools exist on all Linux distros to easily decompress xz. Same was true for bzip2 which came before it, and xz even displaced bzip2 where the latter had superior compression ratio (such as compressing manpages).

Zstd is great as a jack of all trades, I just think the majority of regular people would rather have a smaller file size and spend 3 seconds decompressing it rather than have a larger file size and spend 2 seconds decompressing it in a normal user workflow.

zstd is now getting more widespread as it delivers basically the same compression ratio and same or superior speed as xz, lz4, etc. depending on the compression parameters, and can supersede all of those.

In the enwik9 test, decompression speed was not 2 seconds vs. 3 seconds but rather 2.2 seconds for zstd vs. 20 seconds for xz.
Other LZMA implementations like lzturbo are both faster than xz and offer better compression ratio (with some tradeoff regarding memory use).