Patches Revived For A Zstd-Compressed Linux Kernel While Dropping LZMA & BZIP2

[microcode]

Not entirely true. At work, we're logging proprietary sensor data (basically CAN frames) plus some header plus some more stuff.

Guess what? bzip2 compresses best - better than xz. Files are some 10% smaller. It's quite slow, true, however if the data rate is low, compressing live data on some older ARM is fast enough.

Previously, we used gz as well, but I replaced it with zstd. xz is too slow for our purpose .

Fun fact : if I preprocess the log data with a large burrow-wheeler-transform, xz manages to close the gap to bz2.

Hot damn, I suppose there's always something. I'll have to remember to try bzip2 next time I'm compressing something structured like that.

[hotaru]

why would they remove the fastest one, the only one that allows multithreaded compression and decompression? sure, the implementation in the kernel sucks and only uses a single thread, but the solution is to fix the implementation, not doom everyone to horribly slow single-threaded decompression forever.

[Beherit]

This link details a lot of arguments for why LZMA2 and XZ are poor quality archive formats, and why LZMA1 is superior to them.

http://www.nongnu.org/lzip/xz_inadequate.html

The arguments are very similar to the ARJ vs PKZIP debate in the 90s. ARJ users advocated that ARJ was so much more secure, safe and recoverable than PKZIP. Either way, we all know how that went...

[Imroy]

What support for Zstd compression on BTRFS in GRUB?

[oleid]

BZIP2 is basically useless these days except for compatibility, the decompressor is so slow that it's not appropriate in an embedded setting, and the compression is worse than LZMA2 (and often worse than DEFLATE from zopfli), and outside of an embedded setting, you're often better off with simpler compression anyway due to the capacity and speed of desktop/server storage devices.

Not entirely true. At work, we're logging proprietary sensor data (basically CAN frames) plus some header plus some more stuff.


Guess what? bzip2 compresses best - better than xz. Files are some 10% smaller. It's quite slow, true, however if the data rate is low, compressing live data on some older ARM is fast enough.


Previously, we used gz as well, but I replaced it with zstd. xz is too slow for our purpose .



Fun fact : if I preprocess the log data with a large burrow-wheeler-transform, xz manages to close the gap to bz2.

[microcode]

BZIP2 is basically useless these days except for compatibility, the decompressor is so slow that it's not appropriate in an embedded setting, and the compression is worse than LZMA2 (and often worse than DEFLATE from zopfli), and outside of an embedded setting, you're often better off with simpler compression anyway due to the capacity and speed of desktop/server storage devices.

[EmbraceUnity]

This link details a lot of arguments for why LZMA2 and XZ are poor quality archive formats, and why LZMA1 is superior to them.

http://www.nongnu.org/lzip/xz_inadequate.html

[phoronix]

Phoronix: Patches Revived For A Zstd-Compressed Linux Kernel While Dropping LZMA & BZIP2

For more than a year it's been talked about adding an option to support Zstd-compressed Linux kernel images while it looks like that Facebook-backed high performance compression algorithm for kernel images could soon finally be mainlined...

http://www.phoronix.com/scan.php?page=news_item&px=Linux-Zstd-Kernel-Image-EOY2018