Kernel Bisecting Has Never Been Faster Than With AMD EPYC + AMD Threadripper
Traditionally bisecting every performance regression I come across at Phoronix whether it be in compilers, the Linux kernel, or other code-bases has been prohibitively time consuming (and in turn cooling/power expensive) especially when operating on thin margins as is from the depressed state of the web ad industry, etc, but the incredible speed of the latest EPYC and Threadripper processors -- and having many systems with them around -- has been working wonders for massive time savings and tracking down such issues.
In continuation of yesterday's article looking at Linux 5.0 to 5.9 kernel benchmarking and some regressions, besides the explained I/O difference in Linux 5.9 the big performance issue hurting the Apache web server performance on Linux 5.9 continues to be investigated.
Testing continues and have been discussing the issue with Linus Torvalds and carrying out more tests. While the issue isn't specific to AMD CPUs, it fortunately reproduces fine on not only the EPYC CPUs I have been using for debugging it but also the Ryzen Threadripper 3900 series:
On every system tried, Linux 5.9 depresses the Apache web server performance by wide margins, so thankfully with the EPYC and Threadripper boxes around there is no shortage of hardware for reproducing the problem while enjoying fast kernel build times and in turn greater productivity. Plus the speed of the processors help for turning around the benchmarks as quickly as possible with the Phoronix Test Suite. The issue still stems from the patch pointed out in yesterday's article, but as to precisely why it's happening is still being figured out. This morning during testing I also found that not only is Apache hurt by Linux 5.9 but the Nginx web server also is running slower with the latest kernel code:
Linus mentioned that the kernel test robot pointed out a regression last month from the same commit in question and that time for Hackbench albeit never seemed to be followed up on or explored. So then firing up the Hackbench test profile, indeed able to easily reproduce it too depending upon the process/thread count relative to the system under test:
And, yes, did go ahead and also reproduce this active Linux 5.9 kernel regression as well on the Intel Xeon Platinum 8280 2P box for good measure:
On top of that Linux 5.9 issue, another EPYC box has been working through that TensorFlow Lite regression spotted in yesterday's article as well.
It looks like the Phoronix Test Suite is almost done on bisecting that kernel regression present since Linux 5.5 and still there with Linux 5.9 though with some improvements since 5.5... More details on that soon with the high-core-count hardware wrapping up shortly.
In-progress bisecting of TensorFlow with the Phoronix Test Suite on that aging kernel regression.
For software developers wanting a high-level look at how different CPUs compare for compile speed performance, there are the build-linux-kernel results ranging from the top-end AMD EPYC 7742 2P doing a default configured kernel build in less than 20 seconds to the median build speed of all systems tested being around 87 seconds. Similarly when looking at the LLVM build speed for that large compiler stack the top ranks are all secured by AMD Zen 2 hardware.
And in case you missed it, yesterday's article on the Linux 5.0 to Linux 5.9 performance. And perhaps it's time to start placing bets how quickly Zen 3 processors will be able to build the Linux kernel and other large code-bases?
11 Comments
