Linux 6.2 Aims To Ship Updated Zstd Implementation

[phoronix]

Phoronix: Linux 6.2 Aims To Ship Updated Zstd Implementation

While various Linux kernel components are increasingly making use of the Zstandard compression algorithm, the Zstd code in the kernel has tended to trail behind upstream. Fortunately, a push is underway to get Zstd 1.5.2 in the Linux 6.2 kernel cycle that kicks off at year's end...

https://www.phoronix.com/news/Linux-6.2-Newer-Zstd

[_r00t-]

BTRFS likes that!

[akarypid]

I recall loads of people in uproar over this, when btrfs was updated to include some performance improvements (but the zstd version was held back). Looks like this will cover that gap?

I use btrfs as my main file-system in EndeavourOS, with compression enabled, so I'm curious to see if it actually makes a difference. Based on the amount of complaints, one would think this will improve speed by a lot...

EDIT: Found the thread I was talking about: https://www.phoronix.com/forums/foru...ith-linux-5-19

[discordian]

Doesnt seem like there will be big performance improvements, if fact they had to work to get it back to "neutral":

https://github.com/facebook/zstd/issues/3275

Last time there was a year long gap, dont thing were gonna see comparable jumps again.

[akarypid]

Doesnt seem like there will be big performance improvements, if fact they had to work to get it back to "neutral":

https://github.com/facebook/zstd/issues/3275

Last time there was a year long gap, dont thing were gonna see comparable jumps again.

Well, there you have it! I was so skeptical in the first place, but reading that issue seems to me like they are rushing things...

[skeevy420]

It'd be a lot cooler if file systems had access to Zstd-fast levels for the scenarios where we care about speed over compression. Zstd-fast:500 has LZ4-like performance so you're not bottlenecking your NVMe or ramdisk like you are with Zstd:1.

Don't get me wrong, regular Zstd levels are great for older SSDs, HDDs, archiving data, and mostly read-only data on high-speed drives. They're just not the best for high-speed drives used for active read/write situations. I'm just saying that /tmp, /var, and /home have different needs than /usr, /boot, and /etc.

[mangeek]

Is there some sort of mechanism whereby libraries like this could be easily maintained outside of the kernel proper? It seems to me like a bunch of similar things might benefit from something akin to eBPF. It's a drag that things like Zstd will be 'locked in' at an older version for most of Ubuntu 22.04 and RHEL9's lives unless they're specifically backported. Maybe an optional 'feature update' overlay to LongTerm kernels?

[Danny3]

Finally!
Glad to hear that someone is working on upgrading this to be more in sync with the upstream!
I guess that many of use with limited storage space that use BTRFS+Zstd compression will be happy.
I hope the regression found will be fixed in time for this patch to be merged in Linux 6.2!
Even though I would merge it even with that regression that I'm sure it could be fixed in the RCs or minor point releases.

[akarypid]

It'd be a lot cooler if file systems had access to Zstd-fast levels for the scenarios where we care about speed over compression. Zstd-fast:500 has LZ4-like performance so you're not bottlenecking your NVMe or ramdisk like you are with Zstd:1.

Don't get me wrong, regular Zstd levels are great for older SSDs, HDDs, archiving data, and mostly read-only data on high-speed drives. They're just not the best for high-speed drives used for active read/write situations. I'm just saying that /tmp, /var, and /home have different needs than /usr, /boot, and /etc.

That's an interesting take. It dawns on me now, that the general expectation used to be that CPU speeds will increase faster than storage speeds. NVMe and the latest PCIe versions have turned this on its head recently!

In the age of mechanical drives, there was in fact a period when this sort of compression would actually speed things up! This was because the CPU time spent to do this type of thing, resulted in reduced access time for storage, which would net positively after all was done!

[skeevy420]

That's an interesting take. It dawns on me now, that the general expectation used to be that CPU speeds will increase faster than storage speeds. NVMe and the latest PCIe versions have turned this on its head recently!

In the age of mechanical drives, there was in fact a period when this sort of compression would actually speed things up! This was because the CPU time spent to do this type of thing, resulted in reduced access time for storage, which would net positively after all was done!

Exactly, but what's key to keep in mind is that with Zstd you're really only ever hit on the compression stage. Decompression should be blazingly fast regardless of compression level used. That's why I use Zstd:19 for all my basically read-only data on ZFS like receipts and game data. Game save data, Wine prefixes, etc go on an LZ4 volume (there's no point in trying to optimize for Zstd-fast on a 3-disk HDD raid). Having -fast is really only a benefit for a few specific places; and some of those places already have LZ4 support making it somewhat moot...but it'd be nice to have none the less.

Having the kernel with both Zstd and Zstd-fast but not being able to really utilize -fast is like if the Linux kernel was a Gerber Multi Tool with the pliers rusted shut.