The Bizarre Case Of Zstd's Very Slow Performance On Arch Linux

[Beherit]

Why is no one asking why the c99 flag was added, which is an older standard than both c11 and c18, when it’s obviously not required to compile zstd?

Is this some kind of cmake or arch linux standard procedure done to all packages? Just trying to understand the reason behind it.

[Vorpal]

Why is no one asking why the c99 flag was added, which is an older standard than both c11 and c18, when it’s obviously not required to compile zstd?

Is this some kind of cmake or arch linux standard procedure done to all packages? Just trying to understand the reason behind it.

Definitely not Arch, they are just using a build system provided by upstream. I have no interest in digging into this further, but feel free to look into the git history and report back here!

[CochainComplex]

A prime example illustrtating why Michaels work is important even beyond "for the enthusiastic nerds" who just want to pick the distro having 5% above average performance.
Putting Distros in a Meta comparesion does reveal such "small" issues or general regressions. For me its still suprising why there is no QA instance performing such benchmarks daily with the latest snapshot on the distromaintainer / kernel maintainer side? I guess clear linux might be an exception here. But it could be so easy.

[arQon]

It is very surprising that C99 could have such an impact. I need to research on "restrict" and check some older tools. I always assumed "newer compiler makes the binary go brrrr", but maybe it is worth updating the C standard also more than we thought?

IDK who "we" are in that last sentence, but anyone not using at least C99 from the moment the compilers were stable has either an albatross codebase or a clueless technical director. (And in either case, you have my condolences. )

> I just hope it is not something that could regress with other versions.

Not really, no. "restrict" is such a non-event in practice, and all it does is give the compiler a hint, just like "register" used to in the early days.

Realistically though, restrict is exceptionally unlikely to actually buy you anything: one or two saved loads/stores out of thousands, in a handful of functions at best - it's that sort of scale. LZ compressors though, ARE once such exception: if you're going to re-process the same data thousands of times, and the compiler is adding fences etc to keep your code honest, that would definitely not be good for performance.

Obv the real answer here is that all Arch users should buy 5800X's.

[uxmkt]

This is just a measurement error in the zstd benchmark mode caused by missing timespec support and not actually a performance issue

So, one has to wonder why Phoronix relies on programs reporting their own time rather than just slapping /bin/time in front of it and trusting but the wall clock.

[Vermilion]

Let's face it, it's always something with Arch. When has it ever performed well in a multi-distro benchmark?

Really? I mean, even in this specific benchmark it consistently comes in the top 3, aside from the zstd compiling bug.

I can't recall a single time in all these years.

Maybe check last year's benchmark? Not that a 1% above average in benchmarks decides how well distros perform in real usage.

[discordian]

Silly, yes. Inefficient, absolutely not: given an mmap/fd/descriptor, you could easily do an entire stream in a single call; or blocks of 64K at a time, or etc etc.

It's a very BAD idea, certainly, especially for something that's as in flux as zstd; but that doesn't mean it wouldn't be *performant*.

The crypto interface already allows this, you can feed data in/out via a Unix socket afair.

[discordian]

Most likely the issue is c11 vs gnu11. Don't even start why someone is using explicit compiler flags for the standard with a competent build system like cmake.

[graysky]

https://bugs.archlinux.org/task/75104

[arQon]

Most likely the issue is c11 vs gnu11. Don't even start why someone is using explicit compiler flags for the standard with a competent build system like cmake.

1) Calling CMake "competent" is kind of you, but has never been my experience with it.
2) Nearly *every* build uses custom flags. It would be typical to just add them to that list.