Fwupd Switches From XZ To Zstd Compression: More Trust & Slightly Better Performance

[piotrj3]

If you restrict 7zip to the xz format it can't really do much better, the algorithm is roughly the same, xz just has lower memory limits for it's dictionary.
xz was a fork because in the early days 7zip couldn't stream/pipe (or something in that vain).

For zstd you can even get much better results with long mode: zstd --ultra -22 --long=31 but it might not be compatible with all decompressor implementations.

I think result differences comes from alignment. Because even when trying to compress something very large (like 2GB+ in size) differences are still below 4KB size. Thing is over the years Igor Pavlov improved a ton performance of LZMA, like for years you had patch notes of sort improved performance of that improved performance of that. It accumulated.

[Anux]

Yes, the performance is better, I meant compression wise it can't do much better while using the xz format, it can with .7zip though.

[Weasel]

Btw, Facebook is also a large contributor to btrfs file system. By this logic, maybe we all should avoid btrfs like a plague?

I thought that was already a given.

[Weasel]

Would i trust ZSTD over random maintainer with unknown indentity, probably from China who only exist in project a little over 2 years? Yes for sure.

What if that person had proven indentity? Like you know attended conferences etc. - That i could debate.

Seriously biggest trust has creator of LZMA Igor Pavlov, but linux world decided to fork from him long time ago. You know guy only thanklessly maintained project for 25 years and made certain companies like WinRAR a joke.

Oh, I wasn't talking about that filth Jia Tan, but the original maintainer (Lasse Collin).

[gbcox]

I would assume many projects are going to switch to zstd. The reason has been discussed many times before, but due to this latest exploit, people are finally paying attention. It's called the Nebraska problem: https://www.explainxkcd.com/wiki/ind...47:_Dependency

[qarium]

It's a sad story. Lasse Collin developed and maintained Xz as an open source compression tool for well over a decade until mental health issues set in. His reward is to be remembered as the guy who allowed a malicious actor to introduce a backdoor into the software which could easily not have been caught. Not a great advertisement for being an open source maintainer.

you surly do understand that this could happen to anyone even to people who clearly have no mental health issues and who surly would never allow malicious actors to do something like this.

it can happen to anyone because software code is complex and you can use complexity to perform obfuscation...

ok you will never change your mind about that but Lasse Collin is innocent always remember this.

[Robust0522]

you surly do understand that this could happen to anyone even to people who clearly have no mental health issues and who surly would never allow malicious actors to do something like this.

it can happen to anyone because software code is complex and you can use complexity to perform obfuscation...

ok you will never change your mind about that but Lasse Collin is innocent always remember this.

I didn't say that it couldn't. Though this case, mental health absolutely played a role in his being manipulated into handing the keys over to the malicious actor.

My point was simply that being an open source maintainer is in the majority of cases, a rotten deal. These stories of burnout and mental health are extremely common; a great proportion of projects, including those relied on by the most profitable organisations on the planet, are maintained for free by a single person.

Lasse Collin is indeed innocent, but the fact is that people will remember this incident and not the decade of work and stewardship over a widely used open source project. That's the sad reality.

For open source to truly thrive, we need to transition to a culture in which people feel obligated to give back to those projects which have helped them (and who can give). While there are individuals who definitely feel that way, I would say the culture as a whole, both on the personal and corporate side, is moving in the opposite direction.

[ant8]

You are a despicable and pitiful being.

Throwing a dedicated, talented and struggling developer together with a malicious, year-long planned, targeted and potentially state-actor based attack is not only wrong, but also obnoxious fud.

(I agree with Theo on that one: https://www.youtube.com/watch?v=0pT-dWpmwhA)

That being said, it is nice to see zstd adoption.

Thanks for sharing the video. It was a really good and fascinating video.

[MorrisS.]

XZ has been affected by some kind of hack.

[qarium]

I actually tested 7za (23.01 version from opensuse repository) vs xz-utils downgraded from default repository.
I compressed some old thing i had soapui 5.7.0 around, installed as entire directory, tarred and after compressed with 7za and with xz. Both create .xz format.
Compression size at -mx9 (or -9 in case of xz) is pretty much indentical (I assume 7za does a little bigger alligment, but size difference is 0.0023% of one to another, so negligable, and few more compression tests shown same thing size is not the same but almost exactly the same).
Biggest difference is performance. At same amount of threads used, 7za heavly outperforms xz.
time xz -z -k -9 -T16 soap.tar
real 0m52.695s
user 1m29.057s
sys 0m0.410s
time 7za a -txz -mx=9 -m0=lzma2 soap.xz soap.tar
(some 7za verbose stuff, it said it used 16threads automatically)
real 0m29.788s
user 0m53.919s
sys 0m0.609s
I compared few more other stuff bigger and smaller and stuff that don't compress well (like mp4 file) and same thing happens, size is pretty much the same, while 7za heavly outperform xz speedwise.
Another test took xz-utils compressed soap.tar.xz file (not 7za).
And told both to decompress.
time xz -d -k -T16 soapold.tar.xz
real 0m3.001s
user 0m5.642s
sys 0m0.271s
time 7za x soapold.tar.xz
(some verbose stuff from 7za)
real 0m1.736s
user 0m2.908s
sys 0m0.240s
Literally i don't see a reason to use xz-utils on linux.
Edit added ZSTD:
time zstd -T16 --ultra -22 soap.tar -o archive.tar.zst
real 1m28.745s
user 1m28.471s
sys 0m0.460s

time zstd -d archive.tar.zst -o archive.tar
real 0m0.269s
user 0m0.264s
sys 0m0.147s
Overall sizes:
315023360 soap.tar
158759424 soapold.tar.xz (from xz utils)
158763196 soap7z.xz (7za doing xz archive on tar)
162068083 archive.tar.zst
Of course zstd decompression speed is unmatched, but 7za's 50-100% performance gain relative to xz-utils is literally for free to take around, and on top of that free from backdoors and having actual maintainer.

this is really great news looks like 7za is the current tech and xz is abandonware.

zstd is also nice of course. but with your data i would say no one should use xz anymore and use 7za instead.