Linux 5.16's New Cluster Scheduling Is Causing Regression, Further Hurting Alder Lake

Linux 5.16-rc1 is coming out later today and already I'm seeing some fallout in the new kernel's performance... In particular, bad news for Alder Lake that is already seeing the Linux performance trailing Windows 11 seemingly due to the lack of Thread Director integration right now in the kernel and any other missing optimizations around Intel's hybrid architecture. A new feature of Linux 5.16 is unfortunately having unintended regressions for Alder Lake with at least the flagship Core i9 12900K. Here are the results from the latest kernel bisecting that uncovered this latest upstream slowdown.

On the hunt of the first of some performance changes being explored with the in-development Linux 5.16...

With the Intel Alder Lake benchmarks I've been delivering so far this month with the Core i5 12600K and Core i9 12900K it's been with Linux 5.15 or earlier given the Linux 5.16 Git merge window has been happening. But as things slowed down for the merge window, I've been trying out Alder Lake on Linux 5.16. There isn't any Thread Director patches or other big features specific for ADL-S in Linux 5.16 besides finally having the graphics support enabled by default. But, unfortunately, that early testing showed Linux 5.16 performing worse than 5.15...

Initial Linux 5.16 vs. 5.15 figures were shown in this past week's Windows 11 vs. Linux Alder Lake benchmarks where I initially began noticing the Git mainline state being in poor shape for Intel's new wares. I've spent the past few days trying out more systems and ultimately bisecting this still active performance regression on the Linux 5.16 development code. Across a number of different workloads, the Core i9 12900K was consistently performing worse on Linux 5.16 Git than Linux 5.15 and prior kernels:

But it wasn't due to P vs. E core chance luck or so as the run-to-run variance in these cases was low (compared to some of the other previously showcased noisy benchmarks where it was clear ADL-S getting bounced between the Golden Cove and Gracemont)...

Linux 5.13 through 5.15 was quite stable, and then all of a sudden with Linux 5.16 was a measurable performance loss across multiple benchmarks...

Being affected as well were numerous real-world workloads...

So then it was onward to investigating further and bisecting the problem.