Zstd Compression For EROFS Published: Better Than LZ4 But Higher CPU Costs

[Mitch]

Isn't EROFS a read-only filesysrem? I think you wouldn't get a better choice out of current compression algorithms than LZ4_HC. It has a great ratio and great decompress and the only expense is paid up front during writes. It sacrifices only write speed and can compete with ZSTD on ratios.

So it has ratios like ZSTD and read speeds like LZ4. I cannot fathom something more appropriate than that.

[Quackdoc]

Finally, ZSTD is one of the things we have been waiting for Bliss. ZSTD has much greater compression the LZ4. This will help a bit. Bye Bye squashfs

[intelfx]

Also, this way I don't have to live in disgust with myself for having integrated something that bears Zuck symbology.

You know, right, that LZ4 and Zstd were developed (and are maintained) by the same person, Yann Collet?

[geerge]

Imagine if instead of all file systems independently implementing Zstd and LZ4 support that all file systems could just use whatever compression algorithm was present in the kernel.

That's kind of what red hat's VDO does. In a way it's neat and in another way it makes some design decisions that might not be ideal, it's probably not flexible enough in its current state but IMO it has potential. tl;dr it's a man in the middle between a block device and a file system that can do compression and dedupe: https://access.redhat.com/documentat...do-quick-start

[skeevy420]

That's kind of what red hat's VDO does. In a way it's neat and in another way it makes some design decisions that might not be ideal, it's probably not flexible enough in its current state but IMO it has potential. tl;dr it's a man in the middle between a block device and a file system that can do compression and dedupe: https://access.redhat.com/documentat...do-quick-start

Fair assessment. That sums up how I feel about it.

[billli11]

Isn't EROFS a read-only filesysrem? I think you wouldn't get a better choice out of current compression algorithms than LZ4_HC. It has a great ratio and great decompress and the only expense is paid up front during writes. It sacrifices only write speed and can compete with ZSTD on ratios.

So it has ratios like ZSTD and read speeds like LZ4. I cannot fathom something more appropriate than that.

EROFS can already use LZ4HC compressors.

in mkfs.erofs usage

Available compressors are: lz4, lz4hc, lzma, deflate

The current problem is EROFS is only using single thread for compression.

[Mitch]

EROFS can already use LZ4HC compressors.

in mkfs.erofs usage

Available compressors are: lz4, lz4hc, lzma, deflate

The current problem is EROFS is only using single thread for compression.

That is frustrating. But how often would this read-only filesysrem be (re)written to in most cases? Monthly?

I can see a dilemma if system operations take way too long or time out. LZ4HC has very slow writes so that seems possible. It just seems worth it in my view unless that slow write speed is adding up somewhere.

[Anux]

Can anyone confirm this? I used following commands to de/compress

lz4 --favor-decSpeed --fast=5 --no-frame-crc Spielstände.tar Spielstände.tar.5.lz4
time lz4 -d -q --no-frame-crc Spielstände.tar.5.lz4
zstd --fast=5 --auto-threads=logical -T0 --no-check Spielstände.tar -o Spielstände.tar.5.zst
time zstd -d -q --no-check Spielstände.tar.5.zst

and got those results:

--fast=	lz4 size	lz4 time	zstd size	zstd time
1	3390570729	4,031s	3210706973	2,805s
2	3402277093	3,733s	3241491349	2,669s
5	3444741865	3,729s	3357375821	2,918s
50	3847588242	3,718s	4042849004	2,807s

Original file was 4926167040 and time cp needed 3,929s.
LZ4 command line interface 64-bits v1.9.4
Zstandard CLI (64-bit) v1.5.5
Ryzen 7 5700G

Even if you compare same size (lz4 --fast=1 and zstd --fast=5) zstd was clearly ahead. So I'm wondering in which cases lz4 is still faster?

Edit: also zstd was faster on compression time

[ddriver]

Can anyone confirm this?

You may be comparing apples to oranges here. lz4 has a lower compression ratio, and in your test the output size is nearly identical, meaning that you are compressing at different points in the efficiency curve for both compressors. At the sweet spot, lz4 should be 2-4 times faster and produce marginally larger output.

You know, right, that LZ4 and Zstd were developed (and are maintained) by the same person, Yann Collet?

So I am supposed to dislike the guy cuz he was hired by a another lousy guy to be exploited by his lousy company?

My ethical problem was clearly stated to be on the Zuck side of things. And if it was good enough, I'd probably reluctantly use it. Just glad that's not the case

[Anux]

You may be comparing apples to oranges here. lz4 has a lower compression ratio, and in your test the output size is nearly identical, meaning that you are compressing at different points in the efficiency curve for both compressors. At the sweet spot, lz4 should be 2-4 times faster and produce marginally larger output.

Where would that sweet spot be? I did a short test with standard levels (left --fast away):
lz4 3380743287 3,803s
zstd 3105589209 2,954s
Also the low anchor with lz4 --best (and of course --favor-decSpeed --no-frame-crc as always):
lz4 3247279715 3,575s

Which in turn means zstd is faster and compresses better. And I checked single threaded zstd decompression (-d -T1), no change. I won't deny that there might be edge cases (another data set?) where lz4 is faster, but clearly not in general anymore.

Edit: best case throughput:
lz4 1314 MB/s
zstd 1760 MB/s
Considering others state 4000 MB/s for lz4, my test might be flawed but my read/write speed seamed to be the limit here:
cp 1195 MB/s
cp of lz4 --best file: 1408 MB/s
dd to dev/null 2000 MB/s
dd bypassing LUKS 2500 MB/s

Puzzling, I thought disk cache should take care of everything, might need to repeat with a RAM disk.

Edit2:
best case throughput in RAM disk
lz4 1999 MB/s
zstd 2642 MB/s