Looking At An Early Performance Regression In Linux 5.13 - Scheduler Related

Since the Linux 5.13 merge window began settling down and especially now with 5.13-rc1 out the door, I've been ramping up performance testing of the Linux 5.13 kernel. So far I've been seeing one area where the kernel is regression and stems from the scheduler changes this cycle.

I'm still early on in the benchmarking process in testing a range of systems with Linux 5.13 compared to 5.12 stable, but from testing on an Intel Core i9 11900K "Rocket Lake" system in particular was a bit intrigued by one of the performance drops on 5.13 that led me to looking closer at it yesterday...


On the Rocket Lake system as well as a Core i5 9400F system (among the few systems tested so far with preliminary benchmarks), the context switching performance as measured by the well known Stress-NG took a nose dive...

But when running ctx-clock for measuring the context switching time, the performance with 5.13 was unchanged. So with the i9-11900K box with Git and the Phoronix Test Suite were used to bisect this dramatic difference in performance on Linux 5.13...

The difference is very noticeable and thus easy to bisect... Linux 5.13-rc1 with Stress-NG context switching is at roughly 70% the performance of 5.12 stable.

Well, up until bisecting at the very end there was some fluctuation in the outcome but those remaining tests all well off their Linux 5.12 performance.

# possible first bad commit: [c722f35b513f807629603bbf24640b1a48be21b5] sched/fair: Bring back select_idle_smt(), but differently
# possible first bad commit: [6db12ee0456d0e369c7b59788d46e15a56ad0294] psi: allow unprivileged users with CAP_SYS_RESOURCE to write psi files
# possible first bad commit: [0a2b65c03e9b47493e1442bf9c84badc60d9bffb] sched/topology: Remove redundant cpumask_and() in init_overlap_sched_group()

The commits in question leading to the drop in Stress-NG context switching performance all point back to scheduler changes introduced for Linux 5.13. That though is where I am at for the moment. Given my limited time and resources, for now firing up more (Phoronix Test Suite automated) benchmarks and on more systems to see what other real-world workloads may be seeing Linux 5.13 performance changes that could also be attributed to those commits or if the scheduler alterations are fairly isolated in their negative impact. Those that appreciate my relentless Linux benchmarking and daily content on Phoronix can show their support by joining Phoronix Premium to help facilitate further testing and bisecting.