The Bizarre Case Of Zstd's Very Slow Performance On Arch Linux

[mercster]

Imagine that, a "cool kid" rolling release distro has trouble keeping things together because they don't do enough QA, because building everything from source as soon as it comes out -funroll-loops -O999999 adds racing stripes to your Linux! Run Fedora or Ubuntu, kids. Trust me.

[Evorster]

Great article. Very informative.

Arch out of the box is not quite the same as the other distros. The developer community is much smaller, and Arch is not really aimed at being the best desktop distro, or server distro or whatever. It's meant to be fun to use, and easy to tinker with, and it keeps things simple. In the hands of a skilled user, it can be anything you want it to be... For some people that is important.

In my case, I installed Arch ten years ago, and am still running that very same install, many computers later, just duplicating it whenever I change hardware. Every now and then, something changes or is updated, and requires a bit of configuration, but for the most part it runs fine and does what I need it to do.

[arQon]

Let's face it, it's always something with Arch. When has it ever performed well in a multi-distro benchmark? I can't recall a single time in all these years.

Damn, dude, you like living dangerously.
There used to be some pretty fanatical Arch users here, and every single article about any other distro, regardless of content or conclusions, was guaranteed to include at least a few "xyz sux, Arch is way better" comments.

"How do you know if someone runs Arch?" is still my favorite Linux joke though. :P

[jacob]

Damn, dude, you like living dangerously.
There used to be some pretty fanatical Arch users here, and every single article about any other distro, regardless of content or conclusions, was guaranteed to include at least a few "xyz sux, Arch is way better" comments.

"How do you know if someone runs Arch?" is still my favorite Linux joke though. :P

Same for BSD

[cb88]

Shouldn't the Zstd' compress / decompress functions be implemented into the Linux kernel or the GNU C library and they only be called by the desktop environment / file manager/ programs / games when need it?

Why is it at the distro level?

Now I wonder why after almost half a year after the last Zstd release the Zstd code in the Linux kernel still has not been upgraded to it...
They said last time that they made it easier to keep it in sync with the upstream Zstd releases, but it looks like they abandoned the idea and the Zstd in the kernel is outdated again.

No absolutely not... ZSTD in the kernel is for when you want to do compressing in kernel space without leaving kernel space, ZSTD outside the kenrel is just that just another userspace program.

Calling into the kernel to do compression would be pretty silly and probably inefficient.

the same goes for encryption... typically you don't have the kernel do that either... the exception being the kernel handles encryption for other things it also is doing in kernel space (in kernel filesystem encryption etc... network filesystems etc...).

[andyprough]

Damn, dude, you like living dangerously.
There used to be some pretty fanatical Arch users here, and every single article about any other distro, regardless of content or conclusions, was guaranteed to include at least a few "xyz sux, Arch is way better" comments.

"How do you know if someone runs Arch?" is still my favorite Linux joke though. :P

I recall there used to be a group of posters here who were in love with compiling their own kernels on Arch with different compiler flags, and they would argue to the death that they were squeezing out a couple percent more performance from their thousands of hours of work.

[archsway]

Um… is there any evidence of an actual performance problem here? It seems that the time is just being measured in a different way.

This isn't the first time I've seen a bug in time accounting for a multi-threaded program, maybe it's better to stick to time(1) for measuring performance?

[kimixa]

A quick search of the source shows that it keys off c99 (or later) support for some features that can significantly affect code generation - such as "restrict" being used in the xxh hash - without which the generated code will likely be considerably slower as the compiler has to assume the input may alias, so it has to read a single byte at a time and can't optimise it for SIMD, or even just larger than a single byte.

There may also be other similar things, a fair few things check the __STDC_VERSION__ macro

Probably not surprising such things can cause a significant performance impact, but it's a bit of weird unclear path of ending up in that situation.

It also uses a different method of calculating runtime with c11, but as the issue here was "fixed" by c99 then it's probably not the major issue here.

[cloyce]

Turns out there is some sort of optimization level difference.
I'm curious whether this unearths some inefficiency/overhead in the C99 target, paving the way for more generic performance gains.

That'd be nice, but my money is on it being intentional. For example, in C99 mode GCC will disable the use of FMA, even when it's allowed by the standard. That's because the "figure out if it's okay" (-ffp-contract=yes) isn't implemented, so it's the same as -ffp-contract=no. (The default is -ffp-contract=fast, which does FMA if the platform supports it, even if it violates the standard.)

That's almost certainly not the problem with zstd, as it should be mostly integer code, but if it were inhibiting vectorization or something like that, it would have a pretty huge effect.

[jacob]

But just now I tried with `github/facebook/zstd` cmake or meson build result no difference, cannot reproduce in my environment .
so maybe the arch devs should have a dig. I guess gcc will not make C99 slow, or should be found out earlier.

Maybe different GCC version?