Zstd Compressed Linux Kernel Images Proposed Once More

[discordian]

Yeah, get that merged finally ffs. The patches from 3-4 years ago still apply easily and this thing is a completely separate stub that gets dropped from memory after one use.
If you don't use it, it won't affect any bytes in the kernel.

[sdack]

I would very much welcome Zstd (de-)compression on ARM platforms.

[R41N3R]

Waiting for this since forever

[GI_Jack]

zstd was made and actively developed by facebrick.

There are OTHER compression algorithms that are good enough. Fuck facebrick.

[skeevy420]

But isn't LZ4 still slightly faster than ZSTD at decompression?

Not once Zstd is using levels like fast 5000. Zstd is faster and compresses better than LZ4 at those levels (just barely).

[schmidtbag]

While I'll admit that it's probably the sort of data I'm compressing... I've not noticed Zst being "10x" faster to decompress than xz or similar. Maybe 2-3x...

Out of curiosity, but what kind of drive are you using? What are you decompressing into? Have you observed what the bottleneck is? I'm genuinely curious, because the 10x faster claim could be a theoretical maximum, maybe on a beefy system decompressing into RAM.

[jntesteves]

zstd was made and actively developed by facebrick.

There are OTHER compression algorithms that are good enough. Fuck facebrick.

Zstd, lz4, xxhash and FSE are creations of Yann Collet. I think zstd was actually already done when he was hired by Facebook.

[Paradigm Shifter]

Out of curiosity, but what kind of drive are you using? What are you decompressing into? Have you observed what the bottleneck is? I'm genuinely curious, because the 10x faster claim could be a theoretical maximum, maybe on a beefy system decompressing into RAM.

Decompressing from a backup drive (Toshiba 12TB M07 Enterprise HDD on SATA bus, in my experience they read/write about 180-190MB/s if all the data is contiguous) to a SATA SSD in the same system (which tops out around 520MB/s read and 450MB/s write). CPU is a Threadripper 2990WX with 128GB of RAM. Zst compiled from source, 1.4.5.

I don't notice the SSD maxing out writes, so I don't think it's that. I didn't really pay attention to CPU usage (or it was as expected, so can't remember). I need to spend some time doing further compression benchmarks for several datasets, so when I have time I'll do so and report back more. Do you have any suggestions for metrics to (attempt to) monitor?

[terrelln]

Decompressing from a backup drive (Toshiba 12TB M07 Enterprise HDD on SATA bus, in my experience they read/write about 180-190MB/s if all the data is contiguous) to a SATA SSD in the same system (which tops out around 520MB/s read and 450MB/s write). CPU is a Threadripper 2990WX with 128GB of RAM. Zst compiled from source, 1.4.5.

I don't notice the SSD maxing out writes, so I don't think it's that. I didn't really pay attention to CPU usage (or it was as expected, so can't remember). I need to spend some time doing further compression benchmarks for several datasets, so when I have time I'll do so and report back more. Do you have any suggestions for metrics to (attempt to) monitor?

I recommend using turbobench to measure compression algorithm performance https://github.com/powturbo/TurboBench. It measures RAM to RAM (de)compression performance. On my Intel i9-9900K zstd decompresses at over 1 GB/s, so the SSD write speed will limit speed. Here is the output of turbobench on my system.

./turbobench -elz4,1,12/zstd,-1,1,3,19/lzma,1,6,9 cantrbry.tar

TurboBench: - Tue Mar 17 19:42:04 2020

C Size ratio% C MB/s D MB/s Name File
483940 17.2 4.86 153.01 lzma 9 cantrbry.tar
485598 17.2 5.33 152.94 lzma 6 cantrbry.tar
516748 18.3 4.86 1042.16 zstd 19 cantrbry.tar
571805 20.3 36.78 134.11 lzma 1 cantrbry.tar
642530 22.8 307.21 1235.17 zstd 3 cantrbry.tar
701401 24.9 481.83 1399.37 zstd 1 cantrbry.tar
899707 31.9 526.23 1576.93 zstd -1 cantrbry.tar
930641 33.0 3.22 2286.16 lz4 12 cantrbry.tar
1229060 43.6 659.45 2316.19 lz4 1 cantrbry.tar

[Paradigm Shifter]

I recommend using turbobench to measure compression algorithm performance https://github.com/powturbo/TurboBench. It measures RAM to RAM (de)compression performance. On my Intel i9-9900K zstd decompresses at over 1 GB/s, so the SSD write speed will limit speed. Here is the output of turbobench on my system.

Thanks, I'll give it a go for reporting something back.

I am really only interested in real-world performance on the type of data I need to (de)compress. Amazing results in benchmarks are wonderful, but I've optimised for benchmarks in the past only to be disappointed with performance in my daily (mission critical) applications. So now I mostly optimise for FFTW, as the faster I can get that to run, the faster 99% of the mathematics I need run.