Running Linux 5.15-rc1 Causing A New Slowdown... Here's A Look

As usual when the Linux 5.15 merge window began wrapping up, I set out to dive into its performance to see what is in store for this next version of the kernel and whether there was any regressions or other performance changes worth noting. Linux 5.15 overall has been in good shape for the "-rc1" state except noticing that code compilation workloads were taking longer on multiple Linux 5.15-rc1-running systems than Linux 5.14 or prior. Seeing it across multiple systems and a very real-world regression, it was worth bisecting and looking closer so here are the details.

It's kernel performance regression time...

Linux 5.15-rc1 performance overall has been looking good at the assortment of systems I have tested so far this week. The performance overall has been inline with expectations and jiving well with the many new Linux 5.15 features. But it quickly became apparent that something was wrong with compiler performance when running on Linux 5.15... Not the speed to compile the kernel, but rather the performance of building other codebases while the system is running Linux 5.15-rc1. This slowdown for build tests was happening for multiple codebases of very real-world and relevant projects and on multiple systems, making it an interesting regression to look at and worth bisecting for an article.

One of the systems tested was a Dell XPS 9310 with Intel Core i7 1165G7 Tiger Lake notebook... Not something you'd want to use for repeated code compilation unless it's your daily dev machine, but the compilation tests were just some of many carried out.

On this Tiger Lake notebook, it quickly became evident that build tests were taking longer on Linux 5.15 compared to 5.14 or 5.13.

But across dozens of other workloads, the Linux 5.15-rc1 performance was flat with the code compilation benchmarks being one of the main outliers.

This wasn't specific to Tiger Lake or caused by using a notebook, as at the other side of the Intel spectrum with the Intel Core i9 11900K "Rocket Lake" desktop the situation was similar. There wasn't much to see out of Rocket Lake's performance on Linux 5.15 but again seeing various build tests taking longer to complete on 5.15-rc1 than prior kernels:

Given more power to the i9-11900K, more benchmarks were ran and the common theme was code compilation taking longer... With this greater set of build tests, it was also interesting to note this Linux 5.15 slowdown impacting not just projects relying on the long-standing GNU Make but also when using build systems like Ninja and Meson.