The Huge Disaster Within The Linux 2.6.35 Kernel

For the past six months we have been monitoring the performance of the very latest Linux kernel code on a daily basis across multiple systems. We have spotted a few regressions -- both positive and negative -- on occasion using our automated daily testing of the Linux kernel, but nothing like what we have encountered the past few days: the Linux 2.6.35 kernel performance has fallen hard. In fact, the performance has fallen very hard in a number of tests and right now, we would consider it a disaster. While the 2.6.35 code has not even seen its first release candidate yet, there are some massive performance drops in a variety of different tests that have yet to be corrected and nothing like we have encountered with previous kernel release cycles especially for a regression that has lived now for about one week.

To cut to the chase, between the 22nd and 24th of May there looks to be at least one commit (though perhaps multiple based upon the different data) within the Linus Torvalds 2.6 Git tree that are negatively affecting many different server/desktop benchmarks. We waited nearly a week to see if these regressions would be organically caught and addressed, but they have not been at least of the Linux 2.6 Git state as of 2010-05-26.

On the test side we have been running the same exact tests everyday on the same systems that are solely dedicated to tracking the performance of the Linux kernel. This is in addition to our Ubuntu Performance Tracker (found at ubuntu-tracker.phoromatic.com) that monitors the performance of the Ubuntu development packages on a daily basis. At no single time since launching this kernel tracker back in late November have we encountered such a large regression touching so many different tests.

You can see all of these results for yourself by visiting kernel-tracker.phoromatic.com. There is 50+ tests being run on a daily basis and right now this testing is being done on two independent systems. Linux kernel test results via the Phoromatic Tracker can be viewed back to its inception last November, but for these purposes today just set the view range to at least the past seven days (beyond 2010-05-21) so that this huge performance drop can be spotted. Included in this article are just a few of the test results while showing their daily test values as far back as possible. One of the systems is the CompuLab Fit-PC2 with an Intel Atom Z520 processor while its hard drive has a Btrfs partition for where the tests are executed. The other system is an ASRock NetTop ION 330 that is built upon NVIDIA's ION platform with a dual-core Intel Atom 330 and is using 64-bit Linux along with a standard EXT4 file-system. Later in this article we also have our results when we manually confirmed this serious Linux performance issue.

As you can see from the first graph above, the performance in our Apache test profile has absolutely been clobbered in recent days on the Atom Z530 system with the Btrfs file-system. The 64-bit Atom 330 NetTop with EXT4 has fallen too, but not by nearly as much. The Poulsbo system goes from sustaining just under 1,900 requests per second the past few weeks to now the past few days floating just over 1,000 requests per second.