8-Way Linux Distribution Comparison On A Dual Xeon Scalable Gold Server

[stiiixy]

Thanks as always for your reporting Fixed now. Pretty long day and looks like I didn't pay enough attention to LibreOffice spell checker today :/

Hire yourself a handfull of volunteer editors to vett your articles quickly before submission.

Might even be a workable idea for future revenue's sake to have some people help translate some of your major articles to some of the more regular non-English languages like Mandarin, Russian, German, French, Portuguese. Food for thought.

Nice article as well. Definitetly some interesting results. Curious as to why Antergos over Manjaro (eeney meeneyed it?) or even Arch (Your results required a GUI)

[Guest]

Debian is targeted to many platforms and some of them are low end so some optimization have to be done when building Debian. Other distributions have more bloatware too. You can make Debian even faster by using Xfce, installing wicd and qasmixer and removing unused systemd services, pulseaudio and networkmanager. Create a non debug 1000Hz timer kernel too, that speeds booting several seconds and applications runs faster.

By removing systemD you'll make it slower actually. Other things you mentioned are valid for any other distribution. Ubuntu has some debug enabled, so I'm using custom kernel.

[linuxgeex]

Why is Debian so fast?

Differences are related to package version, filesystem / mount options, kernel version, library version, compiler version, default machine arch, and CFLAGs.

Each benchmark performs best with a combination of those criteria. Debian aims to perform well on a lowest common denominator platform. That rarely works out ideally for HEDT or leading model multi-socket servers, so it is a bit surprising.

I've had this conversation with lots of people lots of times though. I'll posit this little bit of speculation: I think that modern x86 processors are getting so good at out-of-order execution and branch prediction. I think that even on processors with >256K of L2, SMT is cutting the L2 cache nearly in half for heavily-threaded benchmarks and so unrolling loops is pushing some workloads out of the L2 cache, even on modern x86 cores. So the conservative -O2 CFLAGS setting is doing a lot better than expected. But that won't be the only thing. Part of it will be the compiler version - for some workloads the newer GCC regresses - the Kernel version with similar reason, and possibly the filesystem, and occasionally the package version.

There's some cases where Clear's newer libraries, latest compiler, AVX-512 support, come together and kick Debians lowest-common-denominator butt to the curb. My gut tells me that if you used a Debian system and used alien to install some selective libraries and packages, you'd end up with a system that matched or beat Clear for every benchmark except the filesystem or kernel dependent ones, though the big one - pgsql - seemed to already favor Debian.

[linuxgeex]

... Create a non debug 1000Hz timer kernel too, that speeds booting several seconds and applications runs faster.

CONFIG_HZ 1000 doesn't make the kernel faster. It may give an appearance that it is faster because it is smoother and occasionally (but not always) lower latency, but throughput benchmarks will suffer slightly vs tickless kernel. ghb frame rates will be lower with a CONFIG_HZ 1000 kernel, period.

The plus side to 1000 HZ (or 10000 HZ for that matter) is that applications run much smoother. This isbecause the Linux Kernel's scheduler does a pretty poor job of scheduling smoothly with any kind of load. When multiple processes/threads are in the scheduler queue for one core and one of those fails to give up control of the core at the end of the time slice it gets marked "bad" and becomes a candidate for being skipped in the next time slice in order to ensure other tasks get a fair chance at the core. That effectively lowers HZ, making applications run choppy.

If instead the scheduler sliced the time slice up finer and simply gave a smaller time slice to the "bad" task next time, then all tasks could still manage to get serviced within the 100 HZ time slice. But that would require waking the kernel more often, scheduling tasks more often, which would adversely impact cache locality, CPU sleep states which would use more power and make the CPU run hotter while getting less work done so it would throttle sooner, also it takes about 43ms for the CPU to climb from minimum clock to maximum so a lot of the latency gains you might think you are getting from 1000HZ are simply not there because the CPU doesn't ramp up that fast. Lose Lose Lose, Lose.

So that is why the scheduler chooses to run more coarsely instead of raising HZ - so that it can run faster.

By building a kernel with CONFIG_HZ 1000, you are defeating a design that's there for a real reason, not arbitrary or an accident.

What I will say about CONFIG_HZ 1000 is that yes it is really nice and smooth for things like MAME/Snes9x/Dolphin/PCSX2 emulators, or if you like smooth scrolling and lots of UI eye candy, and don't care about your overall system performance.

[PuckPoltergeist]

Read CONFIG_HZ 1000 Help text, period.

You're the best illustration, that reading doesn't help, as long as you don't understand what you read.

[linuxgeex]

CONFIG_HZ 1000 makes kernel faster, test with a Amlogic S912 device, you see a huge difference. Our X4 860K, X4 845 and intel dualcore 3.0Ghz computers boots 3 seconds faster from ssd and in gaming benchmarks you see 1-2 fps increase. Read CONFIG_HZ 1000 Help text, period.

Use CONFIG_PREEMPT_VOLUNTARY, others makes your desktop choppy.

Amlogic S912 is an ARM device. Nowhere in here...

Debian is targeted to many platforms and some of them are low end so some optimization have to be done when building Debian. Other distributions have more bloatware too. You can make Debian even faster by using Xfce, installing wicd and qasmixer and removing unused systemd services, pulseaudio and networkmanager. Create a non debug 1000Hz timer kernel too, that speeds booting several seconds and applications runs faster.

... did you mention that you were changing the topic to a completely different architecture.

Yes simpler architectures can see improvements from raising HZ, MIPS in particular. However this thread isn't about ARM. It's about "8-Way Linux Distribution Comparison On A Dual Xeon Scalable Gold Server".

On the machine being discussed, or any modern x86 machine, raising HZ to 1000 will damage cache locality significantly, reduce performance per watt, and result in reduced FPS. It will run slower, not faster. You are giving out bad advice for Debian users on x86. HZ used to be 1000 on kernel 2.6.12, Linus lowered it for performance reasons, first to 250 HZ and then to 100 HZ. If you think you are getting better performance on x86, I would suggest that you revisit your test methodology and verify your facts again. The only way you could possibly be getting better FPS from a game on x86 would be because of an artifact of a poor (busy-wait) VSYNC algorithm in the game, the higher scheduling rate then would lower latency after the task had been evicted for its poor behaviour, by scheduling it again sooner, as a result of smaller time slices. Tickless operation will allow the program to run longer with less interruptions and less cache thrash.

Are you sure you are running with CONFIG_HZ 1000 and CONFIG_HZ_PERIODIC=y?

[TheOne]

Antergos is crappy. Next time give us Arch benchs!

Antergos uses ArchLinux repos directly, they just added an additional antergos repo for stuff like yaourt, some additional gnome extensions and mostly to provide packages focused for the desktop... So basically Antergos == Archlinux but with a nice installer. It is funny because many people that uses archlinux tend to say it feels a lot faster and smoother than other distros, but these benchmarks show that it can be a placebo effect. Maybe the package manager (pacman) is faster but everything else isn't, maybe they need to tune the PKGBUILD flags to produce faster binaries.

I my self use Antergos because I really like the rolling realease nature of ArchLinux and hate having all my software getting stalled at certain versions (which distros named stable) but in reality they can contain more bugs than latest official releases, I hated the Ubuntu PPA system nightmare. On Debian, well, you can just switch to testing but... It still lags behind.

[arjan_intel]

Why is Debian so fast?

gcc 7.x has some regressions on performance unfortunately.....

[arjan_intel]

The Clear Linux team will be busy today... figuring out why CL isn't so fast on postgres, PHP and blender (and a few others)

[thelongdivider]

The antergos performance is very bad. It makes me wonder about my arch linux install. Any clear explanations as to why?