Patches Revived For A Zstd-Compressed Linux Kernel While Dropping LZMA & BZIP2

[hotaru]

Fastest? Ever tried lz4?

can it be decompressed in parallel on multiple cores?

[rene]

yep, you're welcome! ;-)

[log0]

can it be decompressed in parallel on multiple cores?

Compression by its nature is a serial process. Going multi-threaded will hurt compression ratio.

lz4 decompression beats memcpy. So I'd argue it is fast enough.

[Weasel]

But LZMA1 sometimes compresses better (less redundancy) than xz.

I hope this proposal gets trashed because it's retarded.

[hotaru]

Compression by its nature is a serial process.

not with bzip2. check the lbzip2 implementation.

Going multi-threaded will hurt compression ratio.

not as much as using lz4 will.

if you have a lot of slow cores, a parallel implementation of bzip2 is similar in speed to lz4, but with significantly better compression ratio.

[jntesteves]

not with bzip2. check the lbzip2 implementation.

not as much as using lz4 will.

if you have a lot of slow cores, a parallel implementation of bzip2 is similar in speed to lz4, but with significantly better compression ratio.

You could equaly parallelize the workload with any of these codecs, though they all work sequentially. This isn't a property specific to bzip2.

[hotaru]

You could equaly parallelize the workload with any of these codecs, though they all work sequentially. This isn't a property specific to bzip2.

no, you couldn't. in bzip2, the blocks are compressed independently, so they don't need to be processed sequentially. this is a property that, among the algorithms available for kernel compression, is specific to bzip2.

[ermo]

Does the lz4 implementation include the -hc option or is that a different port. i was never sure. i'd like to see it tested. unless i'm terribly confused, it should provide the smaller file size while still being faster decompression wise. i'll look it into it more when i have time i guess.

Michael these different options for the kernel compression be hard to test through / benchmark ?

Looking at

./scripts/Makefile.lib

the LZ4 compression of the build artifacts is done with

lz4c -l -c1

This is the legacy format and not the new LZ4 stream format, which supports LZ4HC as I understand it. Don't know if there's a patch available for the kernel that makes its LZ4 decoder support the new hc-compatible format.

In terms of improving the kernel boot speed, the couple of papers I've seen both suggest that the combined time for loading and then decompressing a LZ4HC compiled kernel is the fastest method by some margin.

The way I see it, why not cut down the options to either xz, zstd or lz4hc? Drop bzip2, gzip and lzo since they serve little purpose given that xz > bzip2, zstd > gzip and lz4hc > lzo. But that's just my off-the-cuff judgement -- there could very well be people who for some reason need one of the legacy formats.

[hotaru]

The way I see it, why not cut down the options to either xz, zstd or lz4hc? Drop bzip2, gzip and lzo since they serve little purpose given that xz > bzip2, zstd > gzip and lz4hc > lzo. But that's just my off-the-cuff judgement -- there could very well be people who for some reason need one of the legacy formats.

in terms of decompression speed (even single-threaded) bzip2 > xz.

[ermo]

can it be decompressed in parallel on multiple cores?

So far, I've seen two claims of parallel/multithreaded LZ4 implementations. One of them is even listed on lz4's home page.