Fwupd Switches From XZ To Zstd Compression: More Trust & Slightly Better Performance

[ReaperX7]

The same people made contributions to zstd, so how trustworthy it is remains to be seen.

Honestly, I'd rather see lz4 become more widely used. It has the best performance to compression/decompression lossless ratio.

[varikonniemi]

You are a despicable and pitiful being.

Throwing a dedicated, talented and struggling developer together with a malicious, year-long planned, targeted and potentially state-actor based attack is not only wrong, but also obnoxious fud.

(I agree with Theo on that one: https://www.youtube.com/watch?v=0pT-dWpmwhA)

That being said, it is nice to see zstd adoption.

exactly the same way i trust Linus to maintain a safe kernel for me, i need to trust other projects that make up the operating system i use. If your project gets backdoored i cannot trust you any longer. You might not care about your computer, but some do care about theirs. And you are despicable and pitiful for suggesting others must act indifferent like you.

This is not a personal judgement of what kind of person xz maintainer is, but it is a statement of fact that he got in over his head, and the project is not trustworthy any longer, at least for the time being.

[Quackdoc]

Interesting. From my observation zstd compressed worse than xz (both on some "max"ish compression settings) and I was compressing text logs. Did that improve recently in zstd?

different compression algorithms have different fortes

[Alliancemd]

Considering that zstd compresses much better and extracts much faster, this should of been done anyway.

[muncrief]

Of all the problems I've encountered with Linux over the decades this one concerns me most.

It was such a long term deliberate and well planned effort, with so many steps that went unnoticed, that I'm concerned for other projects as well.

Trust is a beautiful thing, and when Linux was young and far less ubiquitous perhaps it was enough.

But now Linux is at the heart of many critical systems and consumer products, and exploiting that trust and attacking from within is an assault strategy that could reap massive profit, and/or security intelligence for individuals and state actors.

And yet I can't think of any all encompassing answer.

By desire and design Linux is an incredibly fragmented project, and of course I'm not only speaking of the kernel but apps as well, and I can't conceive of any practical course of action that would place it all under one security umbrella.

But these are the times when I'm grateful there are so many, far more intelligent, people out there than me. And I trust someone or some group will devise a strategy to deal with our enemies from within.

[SofS]

I really do not understand why other projects that rely on it did not invest further on XZ. Not so much the format, but compilers and maybe even assembly optimizations do help a lot. Look how 7z has improved over the years for instance. Actually, p7zip has also been abandoned, I have just checked Fedora 39 and its 7za is a version from 2016. p7zip, by the way, lost its purpose after 7z got proper native and open source support for Linux.

By the way, thanks to the super old version of 7z being present in current repositories, comparing with the latest is trivial. Just run its integrated benchmark on 7za from p7zip and on 7zz from from upstream in the same machine with the same settings.

[Paradigm Shifter]

A rather dark ponderance...

Everyone is so focussed on "only a state actor could invest the time" for a "project of this scale" but no one thinks that a large corporation, wishing to gain increased control over worldwide, distributed, critical software solution(s) could also invest the time. If you're a multi-billion dollar global corporation with profits larger than the GDP of a number of countries... dropping a hundred grand a year for two or three years is barely noticeable; it could even be passed off as "security research"?

[WileEPyote]

It is in my usage and that's why I asked. Edit: and the short answer is "no"

This is a 100% honest question because I am genuinely curious and honestly don't know the answer. What use cases need every bit of compression, but at the sacrifice of significant speed increases?

[piotrj3]

I really do not understand why other projects that rely on it did not invest further on XZ. Not so much the format, but compilers and maybe even assembly optimizations do help a lot. Look how 7z has improved over the years for instance. Actually, p7zip has also been abandoned, I have just checked Fedora 39 and its 7za is a version from 2016. p7zip, by the way, lost its purpose after 7z got proper native and open source support for Linux.

By the way, thanks to the super old version of 7z being present in current repositories, comparing with the latest is trivial. Just run its integrated benchmark on 7za from p7zip and on 7zz from from upstream in the same machine with the same settings.

I actually tested 7za (23.01 version from opensuse repository) vs xz-utils downgraded from default repository.

I compressed some old thing i had soapui 5.7.0 around, installed as entire directory, tarred and after compressed with 7za and with xz. Both create .xz format.

Compression size at -mx9 (or -9 in case of xz) is pretty much indentical (I assume 7za does a little bigger alligment, but size difference is 0.0023% of one to another, so negligable, and few more compression tests shown same thing size is not the same but almost exactly the same).

Biggest difference is performance. At same amount of threads used, 7za heavly outperforms xz.

time xz -z -k -9 -T16 soap.tar
real 0m52.695s
user 1m29.057s
sys 0m0.410s

time 7za a -txz -mx=9 -m0=lzma2 soap.xz soap.tar
(some 7za verbose stuff, it said it used 16threads automatically)
real 0m29.788s
user 0m53.919s
sys 0m0.609s

I compared few more other stuff bigger and smaller and stuff that don't compress well (like mp4 file) and same thing happens, size is pretty much the same, while 7za heavly outperform xz speedwise.

Another test took xz-utils compressed soap.tar.xz file (not 7za).
And told both to decompress.
time xz -d -k -T16 soapold.tar.xz

real 0m3.001s
user 0m5.642s
sys 0m0.271s

time 7za x soapold.tar.xz
(some verbose stuff from 7za)
real 0m1.736s
user 0m2.908s
sys 0m0.240s

Literally i don't see a reason to use xz-utils on linux.

Edit added ZSTD:
time zstd -T16 --ultra -22 soap.tar -o archive.tar.zst
real 1m28.745s
user 1m28.471s
sys 0m0.460s

time zstd -d archive.tar.zst -o archive.tar
real 0m0.269s
user 0m0.264s
sys 0m0.147s

Overall sizes:
315023360 soap.tar
158759424 soapold.tar.xz (from xz utils)
158763196 soap7z.xz (7za doing xz archive on tar)
162068083 archive.tar.zst

Of course zstd decompression speed is unmatched, but 7za's 50-100% performance gain relative to xz-utils is literally for free to take around, and on top of that free from backdoors and having actual maintainer.

[Anux]

Overall sizes:
315023360 soap.tar
158759424 soapold.tar.xz (from xz utils)
158763196 soap7z.xz (7za doing xz archive on tar)
162068083 archive.tar.zst

If you restrict 7zip to the xz format it can't really do much better, the algorithm is roughly the same, xz just has lower memory limits for it's dictionary.
xz was a fork because in the early days 7zip couldn't stream/pipe (or something in that vain).

For zstd you can even get much better results with long mode: zstd --ultra -22 --long=31 but it might not be compatible with all decompressor implementations.