Zstd 1.4.5 Released With 5~10% Faster Decompression For x86_64, 15~50% For ARM64

[rmfx]

I would like to see some test compared to Rad Oodle kraken/mermaid/selkie.

[pomac]

Uhhh... there has been quite a bit of changes between the core code and the kernel submitted code...

[evergreen]

ZFS is able to integrate zstd source code almost "as is" for their compressed file system, which works in kernel space (at least on FreeBSD).
I wonder why it would cost more effort than that when it comes to Linux.

[yoshi314]

Could not agree more. I find myself having little use of anything else these days. At least when I am free to choose.
Zstd has such a broad coverage of use-cases (within reasonable limits) that I just can't justify using much else.
The parallelized versions rip trough stuff like a hot knife through butter.

you just need to be careful about how you compress is,which is a bit of a trap.

i had used wal-shipping in postgresql with zstd and i was running into OOM issues, despite files being only 16MB each - on decompression. turns out you have to pick zstd's compression parameters very carefully.

[starshipeleven]

ZFS is able to integrate zstd source code almost "as is" for their compressed file system, which works in kernel space (at least on FreeBSD).
I wonder why it would cost more effort than that when it comes to Linux.

ZFS on Linux is a kernel module loaded with DKMS, so it's still running in kernel space like any other kernel module.

And the way they are integrating zstd is by copy-pasting the code in their own kernel module. I'm not sure why the developers for ZoL aka the new ZFS upstream can't do the same.

[geearf]

The version in the kernel is almost 3 years old. I hope they will update soon, see this open issue: https://github.com/facebook/zstd/issues/1638 But the issue is marked as long-term. So it may need some more years.

I looked at a few path in the kernel git:
https://git.kernel.org/pub/scm/linux...zstd?h=v5.6.14
https://git.kernel.org/pub/scm/linux...td.c?h=v5.6.14
https://git.kernel.org/pub/scm/linux...td.h?h=v5.6.14

The most recent update is almost a year old. Are you sure the version in the kernel is 3 years old though?

[svenh]

I looked at a few path in the kernel git:
https://git.kernel.org/pub/scm/linux...zstd?h=v5.6.14
https://git.kernel.org/pub/scm/linux...td.c?h=v5.6.14
https://git.kernel.org/pub/scm/linux...td.h?h=v5.6.14

The most recent update is almost a year old. Are you sure the version in the kernel is 3 years old though?

I think the issue in zstd's github is correct. Thanks for the links to commits. But except for the initial ones from 2017 that introduced the zstd module, they are tiny and not related to algorithmic improvements.

[geearf]

I think the issue in zstd's github is correct. Thanks for the links to commits. But except for the initial ones from 2017 that introduced the zstd module, they are tiny and not related to algorithmic improvements.

I didn't look at the changes so you may be correct.
As for the github issue, I think the fact that it is a longterm one means it's one that will have to be _solved* continuously, but it clearly does not seem that continuous... I wonder if we could somehow patch the kernel ourselves or if it's too complicated to figure out.

[discordian]

I would like to see some test compared to Rad Oodle kraken/mermaid/selkie.

There is a opensource version at https://github.com/powzix/ooz (No idea how they came to be). Test away

Other than that, one author of the Oodles has an interesting blog (doesnt seem overly biased): https://cbloomrants.blogspot.com/search?q=zstd