One Of The Reasons Why Linux 5.5 Can Be Running Slower

Going back to the start of December with the Linux 5.5 merge window we have encountered several significant performance regressions. Over the weeks since we've reproduced the behavior on both Intel and AMD systems along with large and small CPUs. Following some holiday weekend bisecting fun, here is the cause at least partially for the Linux 5.5 slowdowns.

On a number of different systems this month we've seen several regressions in real-world workloads like NPB and Parboil, PostgreSQL, Memcached, RocksDB, and also synthetic tests like the Hackbench scheduler benchmark. Worth noting, as to be explained, all these systems were running Ubuntu Linux.

But even with hitting these regressions on multiple systems, we're now past mid-way through the Linux 5.5 cycle without any solution in place or much fuss on the kernel mailing list... So perhaps it's something not reproducible by the configurations of many upstream developers. Ultimately, yes, that is partially the cause as to be explained. Yet all the Ubuntu daily kernel images have seemingly been affected by the Linux 5.5 lower performance on these multiple systems.

Been on the bisecting hunt for the #Linux 5.5 woes this weekend and looking like there are at least two separate performance regressions at play... Should have the details on at least one of them this evening.

— Phoronix (@phoronix) December 29, 2019

So I spent a portion of the Christmas weekend beginning to dive into these issues, thanks to Git's bisecting and interfacing with the Phoronix Test Suite.

Speeding up the bisecting process a lot was also using an AMD EPYC 7742 2P server, which with 256 threads combined makes for quick kernel builds... And with this server exhibiting some of the Linux 5.5 slowdowns, it made quick work of bisecting and doing the kernel builds on-host.

So here is why Linux 5.5 is slower at least on some systems...