Looking At An Early Performance Regression In Linux 5.13 - Scheduler Related

Written by Michael Larabel in Linux Kernel on 11 May 2021 at 08:49 AM EDT. 18 Comments
Since the Linux 5.13 merge window began settling down and especially now with 5.13-rc1 out the door, I've been ramping up performance testing of the Linux 5.13 kernel. So far I've been seeing one area where the kernel is regression and stems from the scheduler changes this cycle.

I'm still early on in the benchmarking process in testing a range of systems with Linux 5.13 compared to 5.12 stable, but from testing on an Intel Core i9 11900K "Rocket Lake" system in particular was a bit intrigued by one of the performance drops on 5.13 that led me to looking closer at it yesterday...
Linux 5.13 Rocket Lake Intel Performance

Linux 5.13 Rocket Lake Intel Performance

On the Rocket Lake system as well as a Core i5 9400F system (among the few systems tested so far with preliminary benchmarks), the context switching performance as measured by the well known Stress-NG took a nose dive...
Linux 5.13 Rocket Lake Intel Performance

But when running ctx-clock for measuring the context switching time, the performance with 5.13 was unchanged. So with the i9-11900K box with Git and the Phoronix Test Suite were used to bisect this dramatic difference in performance on Linux 5.13...
Linux 5.13 Rocket Lake Intel Performance

The difference is very noticeable and thus easy to bisect... Linux 5.13-rc1 with Stress-NG context switching is at roughly 70% the performance of 5.12 stable.
Linux 5.13 Rocket Lake Intel Performance

Well, up until bisecting at the very end there was some fluctuation in the outcome but those remaining tests all well off their Linux 5.12 performance.
# possible first bad commit: [c722f35b513f807629603bbf24640b1a48be21b5] sched/fair: Bring back select_idle_smt(), but differently
# possible first bad commit: [6db12ee0456d0e369c7b59788d46e15a56ad0294] psi: allow unprivileged users with CAP_SYS_RESOURCE to write psi files
# possible first bad commit: [0a2b65c03e9b47493e1442bf9c84badc60d9bffb] sched/topology: Remove redundant cpumask_and() in init_overlap_sched_group()

The commits in question leading to the drop in Stress-NG context switching performance all point back to scheduler changes introduced for Linux 5.13. That though is where I am at for the moment. Given my limited time and resources, for now firing up more (Phoronix Test Suite automated) benchmarks and on more systems to see what other real-world workloads may be seeing Linux 5.13 performance changes that could also be attributed to those commits or if the scheduler alterations are fairly isolated in their negative impact. Those that appreciate my relentless Linux benchmarking and daily content on Phoronix can show their support by joining Phoronix Premium to help facilitate further testing and bisecting.
Related News
About The Author
Michael Larabel

Michael Larabel is the principal author of Phoronix.com and founded the site in 2004 with a focus on enriching the Linux hardware experience. Michael has written more than 20,000 articles covering the state of Linux hardware support, Linux performance, graphics drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org automated benchmarking software. He can be followed via Twitter, LinkedIn, or contacted via MichaelLarabel.com.

Popular News This Week