Updated Zstd Implementation Merged For Linux 6.2

[piorunz]

AFICT it uses the kernel implementation, so, it will benefit from this upgrade

That's excellent news if true.

[cynic]

Does Btrfs benefit from this also? Meaning filesystem compression uses that updated Zstd, or it has its own zstd built-in into Btrfs code?

AFAICT it uses the kernel implementation, so, it will benefit from this upgrade

[discordian]

I wonder if in-tree filesystems will have problems because of this, since zstd’s constants was changed in 1.5.0 such that the output from the compressor is not the same as it was in older versions. At least in ZFS, it is assumed that compress(decompress(old_compress(data))) = old_compress(data). This makes updating zstd potentially problematic without treating it as a different version of the compressor internally, which is something that ZFS implemented alongside zstd support so that future updates could be done.

If thats true, then ZFS is incompatible with compression. Thats never a viable assumption.

[ryao]

LOL!
Somebody has good observation skills!

But when you have only 119 GiB of storage and not a high-end CPU, your only hope is BTRFS + Zstd compression, but that must not slow down everything down, especially when extracting / compressing files or copy / move folders with lots of things inside.

And if you can install and play a few games that would be great!

But to be able to do all that, considering the storage and CPU limitations, at least you better have the latest and the greatest improvements in BTRFS and Zstd.

BTRFS keep getting them, but Zstd did not.

Now with Linux 6.2 I'm very happy that they both have.

Too bad that 6.1 is the LTS one as 6.2 seems to be really wonderful and the one that makes pretty much everyone happy!

I use ZFS with zstd and a 1M recordsize on my machine for that. I get higher compression than btrfs can provide even with zstd since btrfs is limited to compressing in 64KB blocks at a time, which harms the compression ratio. You can test this by manually using zstd to compress the Linux kernel at different block sizes. You should see zstd compresses it much more efficiently with a 1MB block size than a 64KB block size.

That is not 100% accurate since it does not do padding like ZFS and btrfs do to round up to say the nearest 4KB sector, but it should still do a good job of showing the difference between the two in terms of space savings.

[ryao]

Finally!
I've been asking for this for almost a year.
But weren't the Zstd developers saying that they plan to release another Zstd version this month and then update the kernel to that?
It's a shame they didn't do that.
Like discussed here:
https://github.com/facebook/zstd/iss...ent-1267381027

I wonder if in-tree filesystems will have problems because of this, since zstd’s constants was changed in 1.5.0 such that the output from the compressor is not the same as it was in older versions. At least in ZFS, it is assumed that compress(decompress(old_compress(data))) = old_compress(data). This makes updating zstd potentially problematic without treating it as a different version of the compressor internally, which is something that ZFS implemented alongside zstd support so that future updates could be done.

[piorunz]

Does Btrfs benefit from this also? Meaning filesystem compression uses that updated Zstd, or it has its own zstd built-in into Btrfs code?

[Mitch]

This isnt just hand in hand with BTRFS improvements but also MGLRU. Zram and Zswap will be so much better than ever before in cases where Zstd is the compression algorithm used.

[Danny3]

We know you have!

LOL!
Somebody has good observation skills!

But when you have only 119 GiB of storage and not a high-end CPU, your only hope is BTRFS + Zstd compression, but that must not slow down everything down, especially when extracting / compressing files or copy / move folders with lots of things inside.

And if you can install and play a few games that would be great!

But to be able to do all that, considering the storage and CPU limitations, at least you better have the latest and the greatest improvements in BTRFS and Zstd.

BTRFS keep getting them, but Zstd did not.

Now with Linux 6.2 I'm very happy that they both have.

Too bad that 6.1 is the LTS one as 6.2 seems to be really wonderful and the one that makes pretty much everyone happy!

[geearf]

Speed and compression improvements will be much appreciated!
It would be nice to get a phoronix test of those.

[rene]

thankfully t2 Linux is shipping that for ages already ;-) https://t2sde.org/packages/linux