A Nasty Performance Regression For Some Intel Systems Wound Up In Linux 6.5 Stable

While catching up on my Linux kernel benchmarking across different systems the Intel Core i9 13900K "Raptor Lake" desktop performance has been particularly volatile. Linux 6.5 is running slower for the Core i9 13900K compared to Linux 6.4 for a number of different workloads. Buckle up for a wild ride but the good news is a stable fix is forthcoming.

With Linux 6.6 bringing the EEVDF scheduler and Intel cluster scheduling re-added for hybrid CPUs -- among many other Linux 6.6 features -- I have been eager to test Linux 6.6-rc1 on a number of different systems. I hadn't a chance to test Linux 6.5 much on Raptor Lake so it ended up being a Linux 6.4 vs. 6.5 vs. 6.6-rc1 comparison. But then I was taken back by Linux 6.5 performing awful on my main Raptor Lake test rig.

The same Intel Core i9 13900K system was used for all of the testing. Unfortunately, there was a very unexpected surprise:

From Linux 6.4.14 to Linux 6.5.1 was an apparent regression across many different workloads on this Core i9 13900K desktop... This was quite unexpected as on AMD systems -- desktops, servers, and laptops -- there weren't any glaring Linux 6.5 regressions but rather a positive shift with the AMD P-State EPP default and simply not seeing any major slowdown.

Here's a closer look at those numbers before moving onto the bisect.