Looking At Why Linux 5.0 Is Running Slower For Apache & PostgreSQL On Some Systems

[phoronix]

Phoronix: Looking At Why Linux 5.0 Is Running Slower For Apache & PostgreSQL On Some Systems

Last week I reported on some slowdowns when running on the Linux 5.0 development kernel for both Intel and AMD systems. As a few days passed and the regression didn't seem to be figured out and addressed by upstream, and several inquiries from Phoronix readers, I spent some time looking at some of the slowdowns encountered when running on this bleeding-edge code...

http://www.phoronix.com/scan.php?page=news_item&px=Linux-5.0-Net-Perf-Bisect

[geearf]

That's awesome that AMD sent you such a beast.
Thank to them and to you for the bisections!

[rstrube]

Thanks for taking the time to do this, you're providing invaluable services to the Linux community!

Your reporting on the previous performance regressions with STIBP was also extremely helpful, and I believe was a major contributing factor to getting the issue resolved (by reinstating better defaults).

[UlisesH]

Amazing work as usual, Michael!!

[TheOne]

Michael made my day, thanks to people like him we have such a great community!

[hax0r]

Imagine checking out git of Linux 2.6.20 and working from there up, checking commit by commit and noting down any performance degradations, at the same time checking for major performance improvements. There must have been quite a few times where performance speedups overshadowed multiple smaller regressions. Going commit by commit sequentially should allow for partial recompilation so entire kernel wouldn't have to be rebuilt each time. Would be a fun project.

[Wojcian]

Are you still using Ubuntu kernels? If yes, it doesn't tell much about generic kernel performance.

[clementhk]

This effect made me registered and subscribed for 100 years.
Thanks.

[MaxToTheMax]

Imagine checking out git of Linux 2.6.20 and working from there up, checking commit by commit and noting down any performance degradations, at the same time checking for major performance improvements. There must have been quite a few times where performance speedups overshadowed multiple smaller regressions. Going commit by commit sequentially should allow for partial recompilation so entire kernel wouldn't have to be rebuilt each time. Would be a fun project.

I did some back of the envelope calculations.

In 2007, when 2.6.20 was released, the kernel was getting about 100 commits per day:

https://www.schoenitzer.de/lks/lks_en.html

In 2017, the kernel saw 71552/365 = 196 commits per day.

So let's call it about 150 commits per day on average for this time range.

If it takes 5 minutes to benchmark each kernel (compile, install, reboot, benchmark,) then that's 150*5/60 = 12.5 hours to benchmark 1 day's worth of commits, so you would make it through the commit history at 2x realtime and finish the project in 5 years.

I'm assuming you'd do something like git rebase to flatten out the commit history into a linear timeline, which it currently isn't.

[milkylainen]

It's getting tedious ranting about the lack of performance regression testing in the Linux kernel.
I don't care. If you break performance for a workload such as DB and Apache, you better have a bloody good reason and no other way around the problem.

I vote for revert. Breaking that much default performance on DB and Apache is insane.
Thanks a bunch as usual Michael.