The Bizarre Case Of Zstd's Very Slow Performance On Arch Linux

[archkde]

GNU autotools >>> CMAKE confirmed

Autohell is not involved here. The Makefile is hand-written.

[binarybanana]

Autohell is not involved here. The Makefile is hand-written.

On, I like these. Simple, easy to hack on, fast builds.

[carewolf]

Why is no one asking why the c99 flag was added, which is an older standard than both c11 and c18, when it’s obviously not required to compile zstd?

Is this some kind of cmake or arch linux standard procedure done to all packages? Just trying to understand the reason behind it.

Several years ago compilers defaulted to c89, so setting it to c99 was an improvement. Of course as compiler finally updated their C versions it became a regression instead.

[skeetre]

Just want to note that Linux From Scratch downloads and compiles zstd from github and uses the default make file which only enables the CFLAGS="-std=c99 -Werror -O0" when you use the c99build target.

I'm of two minds here, yes, the benchmark identified a miscalculation in multithreaded time calc, which is good, but also that it makes it sound like Arch has horrible performance for zstd compression when it really doesn't. I think after multiple times of getting these results it would have been good to do a basic timed test to see if the results are accurate.
I don't use Arch, but I do reference their PKGBUILD files a lot when building additional packages or troubleshooting something when building my LFS systems.

Michael - Thanks for what you do here with Phoronix! I appreciate all your individual hard work on providing the latest Linux news and for your work on phoronix-test-suite.

Oops, guess I glanced over it too quick.. I just looked for the c99 and it looked like it was adding it by default, but I see now that you have to specify the c99build and then it adds those options. I wasn't able to actually check a build, but just looked at the Makefile at work. I retract my earlier statement.

[DanielG]

Just want to note that Linux From Scratch downloads and compiles zstd from github and uses the default make file which enables the CFLAGS="-std=c99 -Werror -O0"

Clever - if you compile with -O0 you can as well screw up the benchmarks with -std=c99, because the performance will suck anyway :-p

[evergreen]

> uses the default make file which enables the CFLAGS="-std=c99 -Werror -O0"

Not sure where you got this information, but no, default Makefile doesn't enable any of these flags. It's essentially limited to -O3. No extra warnings, no extra errors, these are for debug builds.

[deusexmachina]

I know Arch is not a source distro but you actually illustrate my point. You implicitly have to trust your distro no matter which one you use. If the distro provides binaries, you can trust that those binaries are good. Of course sometimes the packagers can get it wrong, like here, but it's the exception rather than the rule (as shown in this instance by the fact that all the other ones have good zstd builds). That's obviously no accident, because especially major distros like Ubuntu perform extensive testing *of the binaries they ship*. On a source distro (or with BSD ports etc) there is basically no way to keep that under control.

Why wouldn't reproducibility allow for this? Gentoo, Arch, Debian... They can be made create the same exact binary from the sane exact code. Debian has mostly done this and I thought Arch had already too.

[deusexmachina]

You are missing the point. Distros compile the source code and then test the binary. Except for rare cases when a problem slips through (like here) you know that the binary has been tested and vetted. On source oriented distros where everyone compiles everything differently, there is no QA of the binaries.

There could be with reproducibility. People tend to think it isn't important or useful but it is worth the effort to do what Debian and Arch to some extent, has done to make sure packages are reproducible. Gentoo could do it too.

[microcode]

Imagine that, a "cool kid" rolling release distro has trouble keeping things together because they don't do enough QA, because building everything from source as soon as it comes out -funroll-loops -O999999 adds racing stripes to your Linux! Run Fedora or Ubuntu, kids. Trust me.

This is not what Archlinux is... and "building from source" is what every distro does.