Originally posted by hotaru
View Post
Announcement
Collapse
No announcement yet.
Patches Revived For A Zstd-Compressed Linux Kernel While Dropping LZMA & BZIP2
Collapse
X
-
Originally posted by ermo View Post
So far, I've seen two claims of parallel/multithreaded LZ4 implementations. One of them is even listed on lz4's home page.
Comment
-
Originally posted by hotaru View Post
where? I've seen claims that they exist, but the lz4 utility in my Linux machine obviously doesn't do parallel decompression (never uses more than one core). its compression ratio is also terrible (a lot worse than even gzip), so it's not really useful even if it is faster than bzip2.
---- from the lz4 home page ( https://lz4.github.io/lz4/ )
Compatible CLI versions
Here are a few compatible alternatives to lz4 command line utility :(...)C++11 multi-threads Takayuki Matsuoka https://github.com/t-mat/lz4mt
---- END
The argument is that I'm personally fine with living with a kernel size of 8-10MB instead of 4-5MB if it means the decompression is 3x faster. Keep in mind that I'm decompressing to a ~40-50 MB uncompressed image on a single core of an old Q9400 where this sort of performance delta is actually noticeable. But what's more, ye olde Q9400 is more or less on par with a modern Pentium J5005 quad core 10W CPU. So it's not like you can't buy modern hardware where my scenario can be duplicated.
I'm not asking you to adopt my use case -- I'm just pointing out that, if you desperately need it, a multithreaded/parallel implementation of the lz4 encoder/decoder does in fact exist.
Comment
-
Originally posted by ermo View Post---- from the lz4 home page ( https://lz4.github.io/lz4/ )
Compatible CLI versions
Here are a few compatible alternatives to lz4 command line utility :(...)C++11 multi-threads Takayuki Matsuoka https://github.com/t-mat/lz4mt
---- END
Comment
-
Code:lz4c -l -c1
Code:lz4 -l -12
Comment
-
Just found out that zst (the command line tool) actually supports multi-threaded compression. No big surprise so far.
What I would call "black magic" if I wouldn't be able to verify myself is that the speedup is nearly linear, while the output is bit-exact to the file produced with 1 thread. No splitting into chunks and degrading compression-ratio involved like most other tools do (xz).
I am rather impressed by that.
- Likes 1
Comment
-
Originally posted by poorguy View PostI'm not sure to see the point of multi-threaded decompression of lz4 for a command line utility.
Comment
-
Originally posted by EmbraceUnity View PostThis link details a lot of arguments for why LZMA2 and XZ are poor quality archive formats, and why LZMA1 is superior to them.
Comment
Comment