Linux 5.5 Seeing Some Wild Swings In Performance - Improvements But Also Regressions

While there still is a week to go in the Linux 5.5 merge window with more feature code still landing, due to scheduler changes and other work already having landed, I already started running some Git benchmarks. Linux 5.5 at this stage appears quite volatile with some really nice improvements in some workloads but also regressions in others.

I started off some Linux 5.5 Git benchmarks a few days ago after seeing the scheduler changes land that are rather heavy this cycle and other work. Plus I wanted to test out some new features like the NVMe hwmon thermal reporting.

I started off with a Cascade Lake system just as it was available and curious to see the impact with a large server.

To much surprise, there were much more swings than we are used to seeing between kernel revisions -- especially with no change in mitigations for any security vulnerabilities between Linux 5.4 and 5.5 Git... Some workloads like Rodinia and Parboil were seeing real improvements with Linux 5.5. The Facebook-developed Hackbench Linux kernel scheduler benchmark was also seeing significant improvements with Linux 5.5, likely explaining at least some of the 5.5 benefits coming from the new scheduler code. But there were also cases like PostgreSQL, Memcached, Pennant, and others seeing severe to small regressions where Linux 5.4 was running faster. (See all of this system's data via this OpenBenchmarking.org result file.)

Not used to seeing such large shifts in Linux kernel performance especially with no mitigation changes, I fired up a completely separate system and ran largely the same benchmarks while again comparing Linux 5.4.0 to Linux Git this week.

On this Skylake server, Hackbench was again much faster on Linux 5.5 likely pointing to the scheduler changes explaining the boosts. Parboil and Rodinia continued to see improvements while Memcached, PostgreSQL, and Stress-NG System V message passing tests were among the cases seeing lower performance. So for many of the same workloads, they reproduced on this different Linux server. (All the data via this OpenBenchmarking.org result file.)

Curiosity got the best of me so I also fired up an AMD EPYC 7601 2P Dell PowerEdge server.

Wild swings again. Stress-NG System V Message Passing, PostgreSQL, and some other micro-benchmarks also showed better performance on Linux 5.4 than 5.5 Git... Meanwhile, Hackbench, Rodinia, and Parboil all showed improvements off Linux 5.5 Git. So at least from a quick examination, this AMD server is seeing similar behavior out of Linux 5.5 Git to the Intel Xeon boxes. (The EPYC result file.)

So while only half-way through the merge window, there does appear to be some compelling performance improvements to find with Linux 5.5 at least for larger systems. But at the same time there are some workloads seeing clear pull-backs in performance compared to Linux 5.4 stable. As the Linux 5.5 merge window settles down I'll be carrying out more benchmarks as hadn't expected to see such swings and normally don't between kernel releases at this stage. Those enjoying all the Linux benchmarking found at Phoronix, you can show your support this holiday.