Zstd 1.4 Brings Even Better Compression / Decompression Performance

[milkylainen]

A recent customer had a downloaded operational software in embedded strategy.
U-boot and Linux would boot after which all of the operational software would be downloaded and unpacked.
They had previously used the stupid tar.gz approach after which I replaced it with a stream parallel zstd unpack on a 24-core CPU.
Boy was it way, way faster.
Zstd is a great project in my book.

[sdack]

No, they are still different products....

The LZ4 wikipedia entry says something else: "LZ4 is also implemented in newer Zstd archiver by Yann Collet." (source)

[Compartmentalisation]

A recent customer had a downloaded operational software in embedded strategy.
U-boot and Linux would boot after which all of the operational software would be downloaded and unpacked.
They had previously used the stupid tar.gz approach after which I replaced it with a stream parallel zstd unpack on a 24-core CPU.
Boy was it way, way faster.
Zstd is a great project in my book.

Easily the best thing to come out of Facebook. I've been using it for so many archival related tasks, and specifically use btrfs on disks just to get both storage use down by 50% on average, and simultaneously speed up everything about it, even on SSDs.

[discordian]

The LZ4 wikipedia entry says something else: "LZ4 is also implemented in newer Zstd archiver by Yann Collet." (source)

That means the Zstd archiver tool supports both the zstd format as well as the lz4 format. Both formats are totally distinct however.
Analogous to a Mediaplayer that supports both mp3 and flac.

[speculatrix]

dumb troll

5 people upvoted my comment, presumably because they found it amusing. I suggest you do "yum install sense-of-humour". Other package managers are available. Your mileage may vary. Humour may vary in different geographic regions.

[sdack]

That means the Zstd archiver tool supports both the zstd format as well as the lz4 format. Both formats are totally distinct however.
Analogous to a Mediaplayer that supports both mp3 and flac.

I've checked the source code now. You, too, seem to be just guessing straight out of your arse, mate.

It doesn't produce a compatible byte stream by random chance as you seem to suggest it. Zstd can compress into and decompress from LZ4 streams and it uses the LZ4 library to do so. So, yes, Zstd does in fact integrate LZ4 now.

[discordian]

I've checked the source code now. You, too, seem to be just guessing straight out of your arse, mate.

It doesn't produce a compatible byte stream by random chance as you seem to suggest it. Zstd can compress into and decompress from LZ4 streams and it uses the LZ4 library to do so. So, yes, Zstd does in fact integrate LZ4 now.

Lol, yeah... just as it "integrates" gzip, and lzma.
There is a commandline tool and a compression format, you seem to have big trouble getting that.

*** zstd command line interface 64-bits v1.3.8, by Yann Collet ***
Usage : 
      zstd [args] [FILE(s)] [-o file] 

FILE    : a filename 
          with no FILE, or when FILE is - , read standard input
Arguments : 
 -#     : # compression level (1-19, default: 3) 
 -d     : decompression 
 -D file: use `file` as Dictionary 
 -o file: result stored into `file` (only if 1 input file) 
 -f     : overwrite output without prompting and (de)compress links 
--rm    : remove source file(s) after successful de/compression 
 -k     : preserve source file(s) (default) 
 -h/-H  : display help/long help and exit 

Advanced arguments : 
 -V     : display Version number and exit 
 -v     : verbose mode; specify multiple times to increase verbosity
 -q     : suppress warnings; specify twice to suppress errors too
 -c     : force write to standard output, even if it is the console
 -l     : print information about zstd compressed files 
--ultra : enable levels beyond 19, up to 22 (requires more memory)
--long[=#]: enable long distance matching with given window log (default: 27)
--fast[=#]: switch to ultra fast compression level (default: 1)
--adapt : dynamically adapt compression level to I/O conditions 
 -T#    : spawns # compression threads (default: 1, 0==# cores) 
 -B#    : select size of each job (default: 0==automatic) 
 --rsyncable : compress using a rsync-friendly method (-B sets block size) 
--no-dictID : don't write dictID into header (dictionary compression)
--[no-]check : integrity check (default: enabled) 
 -r     : operate recursively on directories 
--format=zstd : compress files to the .zst format (default) 
--format=gzip : compress files to the .gz format 
--format=xz : compress files to the .xz format 
--format=lzma : compress files to the .lzma format 
--format=lz4 : compress files to the .lz4 format 
--test  : test compressed file integrity 
--[no-]sparse : sparse mode (default: enabled on file, disabled on stdout)
 -M#    : Set a memory usage limit for decompression 
--no-progress : do not display the progress bar 
--      : All arguments after "--" are treated as files 

Dictionary builder : 
--train ## : create a dictionary from a training set of files 
--train-cover[=k=#,d=#,steps=#,split=#] : use the cover algorithm with optional args
--train-fastcover[=k=#,d=#,f=#,steps=#,split=#,accel=#] : use the fast cover algorithm with optional args
--train-legacy[=s=#] : use the legacy algorithm with selectivity (default: 9)
 -o file : `file` is dictionary name (default: dictionary) 
--maxdict=# : limit dictionary to specified size (default: 112640) 
--dictID=# : force dictionary ID to specified value (default: random)

Benchmark arguments : 
 -b#    : benchmark file(s), using # compression level (default: 3) 
 -e#    : test all compression levels from -bX to # (default: 1)
 -i#    : minimum evaluation time in seconds (default: 3s) 
 -B#    : cut file into independent blocks of size # (default: no block)
--priority=rt : set process priority to real-time

[sdack]

Lol, yeah... just as it "integrates" gzip, and lzma.

Yeah, you keep posting the help page. You're not going learn from that.

[Weasel]

Zstd can compress

Learn the distinction between zstd as a format and zstd as the archiver.

7z archiver can uncompress rar files does it mean rar is part of 7z format?

[sdack]

Learn the distinction between zstd as a format and zstd as the archiver.

I don't have to. I've got the source. Thanks for more bullshit though.

And yes, 7zip can uncompress rar files, but it cannot produce them.

LZ4 and Zstd were both developed by Yann Collet. As far as I remember did he write Zstd after LZ4 and he used the same hashing algorithm in Zstd, which he already applied with LZ4. Only did he take it a good deal further with Zstd. People back then were already asking about the integration of LZ4 into Zstd, because it was an obvious thing to do. It actually took a while before he did it. He then only uses the LZ4 library, but doesn't duplicate the entire LZ4 source code. So now Zstd can not only uncompress LZ4 streams, but it can produce them, too.

But hey, you don't have to believe any of this.