Five Years Of Linux Kernel Benchmarks: 2.6.12 Through 2.6.37
While we have conducted studies related to the Linux kernel performance in the past such as benchmarking up to twelve kernel releases, going out the door this morning are the results from the largest-ever Linux kernel comparison conducted at Phoronix, and very likely the largest ever of its kind regardless of source. Every major Linux kernel release from Linux 2.6.12, which was released in mid-2005, up through the latest Linux 2.6.37 development code was tested. This represents the past five years of the Linux kernel and shows how the performance has evolved over the past 25 stable kernel releases and the most recent 2.6.37 development kernel.
Benchmarking 26 kernels was no easy feat with running nearly two dozen tests each time and each test being run multiple times (usually three to five times as a minimum). Fortunately, with the Phoronix Test Suite combined with an Intel Core i7 "Gulftown" made this process much faster, easier, and more reliable than what would otherwise have been possible. A huge thank you goes out to Intel for supplying Phoronix with the Intel Core i7 970, which is their 32nm Gulftown processor with six physical cores plus Hyper Threading to provide a total of 12 threads. The Core i7 970 has 12MB of L3 cache and is clocked at 3.20GHz while having a maximum turbo frequency of 3.46GHz. This is one very fast desktop processor as shown in our Intel Core i7 970 Linux review and more recently within our LLVMpipe Scaling On Gulftown article where the performance of this Intel LGA-1366 CPU was looked at when running Gallium3D's LLVMpipe when enabling 1/2/3/4/5/6/12-threads. While the i7-970 is very fast, it's also very expensive at approximately $900 USD (NewEgg.com and Amazon.com), but it allowed this major Linux kernel comparison to happen in just under a week of constant testing, which is significantly less time than it would have required if using one of the less powerful Intel or AMD CPUs.
With all of this kernel benchmarking being carried out by the Phoronix Test Suite, particularly when using the latest Phoronix Test Suite 3.0 "Iveland" work, benchmarking all of these kernels was not so tedious and it ensures our kernel test results were automated, easily reproducible, and statistically significant with the tests being carried out multiple times and other measures taken plus other recent advancements. As this is also being done with an Iveland snapshot, the new Phoronix Test Suite graphs are being utilized, which provide error bars on graphs where relevant. The set of tests we ran on every major kernel release from Linux 2.6.12 through Linux 2.6.37 Git included GnuPG, Gcrypt, OpenSSL, NASA NAS Parallel Benchmarks, TTSIOD 3D Renderer, C-Ray, Crafty, MAFFT, Himeno, John The Ripper, LAME MP3, 7-Zip, BYTE, Loopback TCP Network Performance, timed Apache compilation, Apache web-server, PostMark, FS-Mark, IOzone, Threaded I/O Tester, and PBZIP2.
In order to go back all the way to the Linux 2.6.12 kernel, which puts us to the era of Ubuntu 5.10, SuSE 9.3, Fedora Core 4, and Mandrake 2006 distributions, we decided to go with Fedora Core 4 as the base operating system. However, in order to run Fedora Core 4 on the Core i7 970 desktop system rather than 2005-era hardware where this benchmarking process would have easily taken weeks, we ran all tests from within a Fedora Core 4 virtual machine. The Core i7 970 host system was running Ubuntu 10.10 64-bit with the Linux 2.6.35 kernel on an ASRock X58 SuperComputer motherboard, 3GB of DDR3 system memory, a NVIDIA GeForce GTX 460, and a 64GB OCZ Vertex SSD. The 64-bit FC4 virtual machine with its stock packages and installation options (aside from disabling SELinux) was using KVM virtualization and had access to all 12 processor threads, 2GB of system memory, and a 32GB disk image. Every major Linux kernel release was built from source while following the same kernel configuration options, build procedure, and user-space. We wanted to go back even further than the Linux 2.6.12 kernel, but our GCC4 build was having issues compiling some of these even older kernel releases. This 64-bit FC4 virtual machine setup to do our Linux kernel benchmarking was the only active process running on the Gulftown test system.