Zstd-Compressing The Linux Kernel Has Been Brought Up Again

[dwagner]

A second is a lot, considering it doesn't take much to set up your system to boot up in <4s on a cheap TLC SSD. Some people don't like to wait for these things.

Then by all means tell them how to wait even less, using S3 sleep instead of rebooting.

[dwagner]

Need motivation? Steve Jobs (an expert motivator) used the analogy of saving lives.

Steven and his apostles could have "saved even more lives" had they removed the requirement to boot their machines so often. Because not booting takes 0 seconds.

[dwagner]

It is really relevant in a lot of cases. Modern laptops can boot up in 5-10s which means a 1s reduction is a massive improvement

But why would I want to boot my laptop more than a few times per year? For laptops, S3 sleep is usually the much more convenient option, and that takes < 0.5s (and does not involve kernel decompression).

[dwagner]

Windows systems can boot from hibernate in about 3 seconds. Why should Linux take longer?

What use case do you think of where hibernation, which to wake up from is way slower than S3-sleep, is preferable?
(If I intend to not use my laptop for a time long enough to drain the battery in S3 sleep, I don't mind doing an actual shutdown.)

[k1e0x]

I don't get it why there was backlash to add Zstd, the developers showed them the numbers and it makes sense to use it in several widely used scenarios. Other obsolete algorithms can be dropped after this lands. So why does it take so long to get something beneficial like this into the Kernel?

There is a trend to always chase the "new shiny" to always say.. oh the Macintosh is out the terminal is obsolete GUI only. Or Plan 9 makes Unix obsolete! Sometimes the new thing really sucks and people don't see the value in the old methods. We don't advance as fast as we think.

So be careful here.. however with this one I agree with you. Zstd or LZ4 sounds more optimal.

[stormcrow]

If shaving off 1s of boot time is relevant for you, then you clearly have a severe stability issue with your operating system.

BTW: (De-)compression algorithms need very thorough testing against security vulnerabilities. If you put such an algorithm into the kernel, make very, very sure that you have tested it with all kinds of random and maliciously crafted input.

Not really. What may take 1s on the latest Intel/AMD x64 hardware, could be much more of a factor (like tens of seconds to minutes) on low power embedded hardware, which is probably the biggest sector using the Linux kernel. It's not about how fast it takes your laptop to boot, um... unless it's like my vintage HP Centrino Duo laptop I use for a terminal, but I don't really care much there either... it's about how fast it takes a Blackfin controller, or a low power ARM32 board (like early R-Pi models) to unpack its kernel before it can even start to boot. Leaving the kernel uncompressed isn't an option either, as non-volatile storage on embedded devices aren't always large enough and/or the boot loader may have kernel size limits.

As for Zstd from a security prospective, you do realize zstd is already in the Linux kernel in different spots as it is. Btrfs uses it as does squashfs. The algorithms (as opposed to the implementation, don't conflate the two) have been around for decades already, the zstd group just put them together in a unique way to produce better performance than previous incarnations. Other examples of zstd being used in critical places would be FreeBSD's version of OpenZFS which also contains zstd compression similar to btrfs and Ubuntu's version of deb packages which uses zstd by default since 18.10.

[Raka555]

Yeah my testing is more focused on compression speed and compression ratio as it relates to backup speed and lz4 has neither of those strengths i.e. for tar + zstd backup tested speeds

I am curious as to what you have decided to use for your backups ?

[tuxd3v]

Ubuntu got it right with lz4. Adding support for Zstd isn't a problem. Going forward, IO will continue to scale faster than CPU which is already the case.

Indeed, lz4 is good on compression times and decompression times.
Its the default for example Oracle Databases, it don't compromise too much the speed, and depending on database content, you could earn a lot with it..

A comparison for several algos

[eva2000]

Wow, very nicely done!
It is exactly the graph I wanted to see. Glad you didn't use log scale for x-axis.
Pitty lz4 was'nt in your test.
I am very surprised at how well pigz did at high speed compression.
Would have been awesome if you had the same graph for decompression speed vs compression ratio. (Edit: Its hard to get a good picture from the data alone)

Yeah my testing is more focused on compression speed and compression ratio as it relates to backup speed and lz4 has neither of those strengths i.e. for tar + zstd backup tested speeds

If shaving off 1s of boot time is relevant for you, then you clearly have a severe stability issue with your operating system.

BTW: (De-)compression algorithms need very thorough testing against security vulnerabilities. If you put such an algorithm into the kernel, make very, very sure that you have tested it with all kinds of random and maliciously crafted input.

A second is a lot, considering it doesn't take much to set up your system to boot up in <4s on a cheap TLC SSD. Some people don't like to wait for these things.