Configurable Zstd Compression Level Support Is Revived For Btrfs

[phoronix]

Phoronix: Configurable Zstd Compression Level Support Is Revived For Btrfs

Since the Linux 4.14 kernel Btrfs has supported Zstd for transparent file-system compression while a revived patch-set would allow that Zstd compression level to become configurable by the end-user...

http://www.phoronix.com/scan.php?page=news_item&px=Btrfs-Zstd-Configurable-Levels

[oleid]

Ah, that's wonderful! According to my tests, zstd's decompression speed varies not too much, depending on the compression level; even for insane levels like 22 it's in line with what Michael reported. It's still fast for embedded ARM devices - I tested it with some Toradex Colibri T20 and AFAIR decompression was still above 100 MiB/s ( I think close to 200 MiB/s) for in memory decompression and always faster than gzip.

So in the end, you could compress static data with the highest levels and still profit from fast access!

[shmerl]

More important than level is a smart heuristic for detecting when to compress and when not. Right now in non forced mode, btrfs isn't being too smart about it, and if first chunks aren't compressed well, it stops compressing the whole file.

[waxhead]

More important than level is a smart heuristic for detecting when to compress and when not. Right now in non forced mode, btrfs isn't being too smart about it, and if first chunks aren't compressed well, it stops compressing the whole file.

Yes that is true, but you also have the compress-force option that can only be applied as a mount option (not with chattr). This ensures that BTRFS does not bail out in case the first blocks are not compressable and instead run all blocks through the compressor. Even with compress-force BTRFS will not store a the block compressed if the compressed data is larger than the uncompressed data which is really a good thing!

[poorguy]

The "automatic compression detector" of btrfs is a total bullshit.
It just kills compression performance, almost at random.
The "detector" itself is very poorly done, and wrongly believes it can determine compressibility from some trivial-to-grab statistics.
That couldn't be more wrong.

Besides, and that's the funny point, modern compressors such as LZ4, LZO and Zstandard
have built-in skippers which know when things are not compressible and just intelligently skip past the bad section.
The only case for this external detector is zlib, but it's wrongly applied to all !

Try it on `zstd`, it skips the incompressible parts at > 1 GB / s !
That's faster than the processing of this "compressibility detector" .

Bottom line : on btrfs, always use "force compress" on any partition with anything else than zlib.

[geearf]

The "automatic compression detector" of btrfs is a total bullshit.
It just kills compression performance, almost at random.
The "detector" itself is very poorly done, and wrongly believes it can determine compressibility from some trivial-to-grab statistics.
That couldn't be more wrong.

Besides, and that's the funny point, modern compressors such as LZ4, LZO and Zstandard
have built-in skippers which know when things are not compressible and just intelligently skip past the bad section.
The only case for this external detector is zlib, but it's wrongly applied to all !

Try it on `zstd`, it skips the incompressible parts at > 1 GB / s !
That's faster than the processing of this "compressibility detector" .

Bottom line : on btrfs, always use "force compress" on any partition with anything else than zlib.

Oh that's very interesting!
I guess when these patches get in I'll recompress everything and force it.

[darkbasic]

More important than level is a smart heuristic for detecting when to compress and when not. Right now in non forced mode, btrfs isn't being too smart about it, and if first chunks aren't compressed well, it stops compressing the whole file.

That's why I always disable that heuristic.