Linux 5.11 Is Regressing Hard For AMD Performance With Schedutil

It's not the Grinch in 2020 that stole Christmas, but the Schedutil CPU frequency scaling governor on the in-development Linux 5.11 kernel that is thrashing performance for AMD Zen 2 and newer. Distributions like Ubuntu, Fedora, and Manjaro are beginning to use CPUFreq Schedutil by default on newer kernels and thus leading to a very bad initial/out-of-the-box experience with the current behavior on the early Linux 5.11 code.

As noted a few days ago, PostgreSQL performance was looking very good on AMD EPYC 7002 servers, Upon further testing PostgreSQL was indeed improving nicely across multiple AMD servers but when looking at more workloads, it began looking like a very mixed bag with some workloads often regressing significantly on Linux 5.11 in its current state as we approach the end of the merge window this weekend.

Some of the Linux 5.11 regressions are painful... (Don't worry, this CPU was already dead.)

Fill The Stockings With Benchmarks

Thus my holiday week quickly shifted into looking closer at the Linux 5.11 performance on AMD systems. My initial hypothesis panned out as the tests in this article are about to confirm, but the often slower Linux 5.11 performance stems from the Schedutil CPU frequency scaling governor often being worse under this in-development kernel. The big change this cycle involving AMD and Schedutil is the AMD frequency invariance support landing. That support or a bad combination with Schedutil is often leading to measurably lower performance on Linux 5.11.

2020: Schedutil Becomes Increasingly Common

Following ARM and Intel configurations transitioning to Schedutil by default since earlier this year instead of "ondemand", most Linux distributions for CPUFreq are now defaulting to "Schedutil" by default including the likes of Ubuntu, Fedora, Manjaro, etc. The Schedutil governor is driven by the kernel's scheduler utilization data for trying to make more informed decisions over ramping up or down the desired CPU performance state. Schedutil is great in theory and viewed by the upstream developers as the future, but until all its quirks are worked out, it's often leaving performance on the table, not delivering any great performance-per-Watt gains, and often coming in short of the "ondemand" governor.

In terms of AMD users running recent distribution kernel images on AMD hardware, recently Schedutil is appearing as the default. This isn't a new concept but going back a while upstream kernel developers have talked of Schedutil potentially replacing the existing CPU scaling governors. As Linux gamers and HPC/server vendors are generally aware, using the "performance" governor will generally deliver the best performance. Indeed that's the case and fortunately switching to that governor on Linux 5.11 isn't introducing any new penalties, but for those relying on Schedutil / seeing it by default, it's a poor initial experience on Linux 5.11.

Linux 5.11 Introduces AMD Frequency Invariance Support

Intel and ARM have supported frequency invariance within the mainline Linux kernel already while with Linux 5.11 the change leading to the lower performance stems from AMD's introduction of it. Frequency invariance is intended to provide for more accurate load tracking and making better frequency scaling decisions. Frequency invariance deals with the matter of tasks appear larger if the CPU is running at a lower frequency compared to a larger frequency. With frequency invariance, the current frequency relative to the maximum possible frequency is taken into account -- or in AMD's case, the ACPI Collaborative Processor Performance Control (CPPC) support of Zen 2 and newer is leveraged for its current/maximum performance states for calculations. The Schedutil governor is the main user of this information.

And Now... Schedutil Sliding In The Wrong Direction

With the ongoing testing since the prior articles and testing Linux 5.10/5.11 under multiple governors, I can confirm this regressing behavior indeed is only exhibited with Schedutil on Linux 5.11.