One Of The Reasons Why Linux 5.5 Can Be Running Slower

While initially the thinking was the regression(s) may have been due to the scheduler load balancing changes with Linux 5.5, it turned out not to be the case. But the culprit does explain partially why there may not have been much upstream attention to these problems and why the regressions weren't bisected and caught sooner.

With the first round of bisecting, the focus was on the Hackbench result given this test was generally seeing multiple times faster performance on Linux 5.4 and older... Very easy test to get started on for this bisecting mission.

The apparent cause of at least some of the Linux 5.5 slowdowns after bisecting turned out to be AppArmor.

In particular, bce4e7e9c45ef97ac1e30b9cb4adc25b5b5a7cfa / 341c1fda5e17156619fb71acfc7082b2669b4b72 appear to be the first bad state of the kernel.

apparmor: reduce rcu_read_lock scope for aa_file_perm mediation
apparmor: make it so work buffers can be allocated from atomic context

Both of these are marked as fixes for the earlier AppArmor work of using a memory pool instead of per-CPU caches.

With AppArmor being most prominently enabled by default on Ubuntu and Debian, it explains why the likes of Red Hat's upstream developers and others weren't affected and thus not quickly jumping on the problem. Likewise, Intel's Linux performance engineers increasingly using Clear Linux and other platforms where AppArmor isn't used or at least on by default.

While Canonical in the past carried a reputation of not contributing significantly to the upstream Linux kernel, this has improved in recent years. In fact, the AppArmor patches in question were from a Canonical engineer. But still without as much upstream development/testing from them, that could explain why they didn't notice the problem sooner.

The kernel configuration I was used for testing was based on their convenient Ubuntu Mainline Kernel PPA archive, which has AppArmor as part of their default kernel configuration. The AppArmor configuration was also at all of its Ubuntu defaults, meaning all Ubuntu users likely would be affected assuming