Zstd 1.5.5 Released With Corruption Fix, Improved Performance In Some Scenarios

[phoronix]

Phoronix: Zstd 1.5.5 Released With Corruption Fix, Improved Performance In Some Scenarios

Just two months after Zstd 1.5.4 was published, Zstd 1.5.5 has been released as the newest version of this Zstandard compression algorithm implementation. Zstd 1.5.5's release is motivated by addressing a rare corruption bug fix but also has various performance optimizations...

https://www.phoronix.com/news/Zstd-1.5.5

[cl333r]

I generally don't like the "rewrite it in Rust" approach but since Zstd is becoming new Zip it could be worthwhile.

[Anux]

The article didn't mention rust or did I miss something?

Anyway, more than double the performance is pretty neat, I thought there is not much left to optimize with zstd.

[discordian]

The article didn't mention rust or did I miss something?

Anyway, more than double the performance is pretty neat, I thought there is not much left to optimize with zstd.

That improvement is some binary patch functionality of the commandline tool. The (de-)compression performance itself (the zstd DSO) is unlikely to see big jumps (atleast on x86/arm where there are already optimizations down to ASM).

[skeevy420]

The article didn't mention rust or did I miss something?

Anyway, more than double the performance is pretty neat, I thought there is not much left to optimize with zstd.

If you scour the Zstd issues you can find this comment from terrelln from a year and a half ago:

We definitely can't promise anything.

I wouldn't be surprised if there is a community-built rust compressor out in the next few years.

I'm also interested in writing a fast & safe rust (de)compressor. But, that would be more of a personal project for me, so I can't promise anything, or give you any dates.

Devil's Advocate: Does that imply Zstd is currently slow and unsafe?

Thanks, C.

[cl333r]

Devil's Advocate: Does that imply Zstd is currently slow and unsafe?

He probably meant that if he writes one, it won't be for demonstration purposes, he'll shoot straight for it becoming a viable alternative.

[Anux]

discordian THX I was wondering what this --patch-from flag was about ...


There is a decompressor allready: https://github.com/KillingSpark/zstd-rs

Also he wrote "fast and safe" not "faster and saver". Although rewriting it in rust would make it's savety proveable at least.

[skeevy420]

He probably meant that if he writes one, it won't be for demonstration purposes, he'll shoot straight for it becoming a viable alternative.

I was just making a joke C's expense. The ole Rust Good, C Bad hurr durr.

Didn't mean to imply anything about anything

If Nick did it I imagine it would be a great tool regardless of the language. He's done a lot of great stuff over the years that myself and lot of us here rely on daily.

[evergreen]

discordian THX I was wondering what this --patch-from flag was about ...
.

In `--patch-from` mode, a "reference" file is used as a "dictionary" to compress another file.

This is particularly efficient when the 2 files are related, say for example a new version.

I just tested compressing Linux6.2.tar source code (~1.3 GB) using Linux 6.1.tar as a reference.
It got compressed to an insanely small ~6 MB.

Of course, the decompression side must also have the same "reference" file to be able to regenerate the target file.

[Anux]

Uhh that sounds like a perfect fit for package management and distribution.