Bisected: The Unfortunate Reason Linux 4.20 Is Running Slower
After running a lot of tests and then bisecting the Linux 4.20 kernel merge window, the reason for the significant slowdowns in the Linux 4.20 kernel for many real-world workloads is now known...
This latest Linux 4.20 testing endeavor started out with seeing the Intel Core i9 performance pulling back in many synthetic and real-world tests. This ranged from Rodinia scientific OpenMP tests taking 30% longer to Java-based DaCapo tests taking up to ~50% more time to complete to code compilation tests taking measurably longer to lower PostgreSQL database server performance to longer Blender3D rendering times. That happened with a Core i9 7960X and Core i9 7980XE test systems while the AMD Threadripper 2990WX performance was unaffected by the Linux 4.20 upgrade.
In some cases this Linux 4.20 slowdown is enough where the Threadripper 2990WX is able to pick up extra wins over the Core i9 7980XE.
When digging through more of my test system data, a set of systems I have running the latest Linux kernel Git benchmarks every other day also saw a significant pullback in performance from the early days of the Linux 4.20 merge window up through the very latest kernel code as of today. Those affected systems weren't high-end HEDT boxes but included a low-end Core i3 7100 as well as a Xeon E5 v3 and Core i7 systems. AMD systems though still didn't appear impacted. Those tests also found workloads like the Smallpt renderer to slowdown significant, PHP performance to take a major dive, and other scientific workloads like HMMer also faced a major setback compared to the current Linux 4.19 stable series.
Bisecting the Linux 4.20 kernel slowdown... The sizable difference during that process.
With seeing clear performance regressions on a number of systems when running the latest Linux 4.20 code, and especially with being able to reproduce it on high-core-count hardware (thus significantly cutting down the kernel build times), this morning I kicked off the kernel bisecting process to see why this new kernel is causing many workloads to run so much slower than Linux 4.19. With the Phoronix Test Suite doing the heavy-lifting, the problematic commit was quickly uncovered.
Going into this testing my thinking was perhaps an Intel P-State CPU frequency scaling driver regression as something that has caused some performance regressions in the past or perhaps a scheduler change. There's also been a lot of Linux 4.20 changes in general that some unintentional regression must have slipped in there somewhere primarily hurting the Intel Linux performance... As a reminder, Linux 4.20 is the biggest kernel release of the year in terms of lines of code changed with more than 354 thousand lines of new code added at the end of October when this merge window opened.
As outlined in the Linux 4.20 feature overview, there are a lot of exciting changes with this kernel. But why is it slower? More work on f!*#(# Spectre!