Linux 4.0 To Linux 4.15 Kernel Benchmarks

[rick-pri]

This is a pretty tame set of scenarios which were tested here.

If you really want to make Intel look bad in terms of benchmarks with a real world horror show (which I experienced on AWS), you go with an Ivy Bridge, or earlier, processor (c3 instances), with a machine which is a hypervisor, you then run a VM with PostgreSQL (an early 9.x version) with a load on the network (15Mb/s should do it over about 30 connections which are constantly cycling) and not enough memory to fit everything into RAM so that the disk gets accessed frequently, and then generate some kernel calls on the other VMs.

You'll see about a 50% reduction in performance and see that latency comes into play so that everything starts to grind to a halt as the TLB is being flushed near constantly.

The decline in the Apache and NGINX benchmark scores is remarkable. What's the story there?

Certain network/disk IO ops require a TLB flush and webservers do a bunch of that serving incoming connections. I remember seeing a whole list of what causes switching to the kernel (which causes a TLB flush and additional latency) somewhere and network and disk IO operations were in there.

[cj.wijtmans]

Yes, just continuing the trend:
https://en.wikipedia.org/wiki/Critic...el_performance
"Citing an internal Intel study that tracked Linux kernel releases, Bottomley said Linux performance had dropped about two per centage points at every release, for a cumulative drop of about 12 per cent over the last ten releases. "Is this a problem?" he asked Linus Torvalds.
-We're getting bloated and huge. Yes, it's a problem"

Looks like i will be using BSD soon if this does not change. Perhaps Linux should start experimenting with having an internal microkernel. I have personally abandoned the linux desktop already because of its messy ecosystem.

[kebabbert]

So server performance is becoming worse.

Yes, just continuing the trend:

https://en.wikipedia.org/wiki/Criticism_of_Linux#Kernel_performance

"Citing an internal Intel study that tracked Linux kernel releases, Bottomley said Linux performance had dropped about two per centage points at every release, for a cumulative drop of about 12 per cent over the last ten releases. "Is this a problem?" he asked Linus Torvalds.
-We're getting bloated and huge. Yes, it's a problem"

[Quppa]

The decline in the Apache and NGINX benchmark scores is remarkable. What's the story there?

[oleyska]

Thanks, very interesting to see these changes over time. I hope some of the kernel developers read these articles and can identify what has happened when changes over time go for lower performance.

If it is not too much work for Michael, it would also be interesting to see a similar analysis for Zen (Ryzen/Threadripper/Epyc). From what I've read there should be some optimizations for Zen beginning with 4.15.

i 2nd this.
but lets wait to 4.17 as 4.10 was first useful zen kernel.

[cj.wijtmans]

So server performance is becoming worse.

[sabriah]

Thanks, very interesting to see these changes over time. I hope some of the kernel developers read these articles and can identify what has happened when changes over time go for lower performance.

If it is not too much work for Michael, it would also be interesting to see a similar analysis for Zen (Ryzen/Threadripper/Epyc). From what I've read there should be some optimizations for Zen beginning with 4.15.

[AdamOne]

Kernel-devs rarely care about boot time. Obviously if something is out of order theyre gonna track it down but its not highest priority to make linux the fastest booter.
Even the 3D apps that are able to offer some type of performance feedback aren't too interesting since theyre not a manual to making linux faster.
There is some hope with vulkan, but thats a couple years away before it takes off and starts making linux more competitive.

[tajjada]

I have a suspicion that it might be some new functionality that newer versions of systemd can make use of, which did not exist in earlier kernel versions. Hence, boot time is slower, as systemd would be initialising and making use of the new features. If my suspicion is true, then older kernels are only faster due to missing functionality. Kinda like OpenGL benchmarks were a lot faster back when Mesa did not have proper MSAA/antialiasing support.

This is just a suspicion though. No idea if true. My memory does not last that far back and I haven't been tracking systemd (I use OpenRC). Does anyone remember if any major software functionality got introduced in kernel 4.6?

[SyXbiT]

The boot time increase in kernel 4.6 seems concerning. Was it known to the community, or did this slip by quietly?
I would hope there are some automated benchmarks to verify performance (including boot time) after each commit. That's the easiest way to catch regressions.