Ubuntu 7.04 to 8.10 Benchmarks: Is Ubuntu Getting Slower?

Computational: The Dhrystone 2 performance within the BYTE Unix Benchmark was also the fastest on Ubuntu 7.04. There was approximately a 20% drop in performance between 7.04 and 7.10 that remained consistent even in the 8.04 and 8.10 releases. Further showing signs of a significant performance regression between Ubuntu 7.04 and 7.10 was the SciMark 2.0 performance. The composite result from SciMark 2.0 had dropped by nearly 30% between Feisty and Gutsy and it too has yet to be recovered. In our eSpeak Speech Engine benchmark, which measures the time to run a text-to-speech engine converting some text to a WAV file, was a 50% performance drop after Ubuntu 7.04 and it continued with 8.04 and 8.10.

Database: In our SQLite test of measuring the time to perform 2,500 SQL inserts, the performance hadn't dropped off after Ubuntu 7.04 but instead after 8.04 LTS. In this performance drop it was over 2.5x slower. Further reiterating signs of a major performance drop following Ubuntu 7.04 was our Tandem XML benchmark where there were similar losses in performance.

Compilation: This may sound like a broken record now, but in our compilation benchmarks we spotted major performance losses following the Feisty Fawn release. It was noticeably slower to compile Apache, PHP, and ImageMagick in the 7.10, 8.04, and 8.10 releases. This certainly reiterates signs of a major performance problem introduced after Ubuntu 7.04 (perhaps with GCC or the kernel) at least with the Intel hardware used by the ThinkPad T60 in our testing.

Encryption: In many of these benchmarks the performance had dropped off after Ubuntu 7.04, but when it came to encrypting a 2GB file with GnuPG there was enhanced performance starting with 7.10 and it continued into 8.04 and 8.10. The OpenSSL RSA 4096-bit performance was flat-lined across all distributions tested. In the Java-powered Bork File Encrypter, the performance had too slowed down.

Disk: Not impacted by any regression after Ubuntu 7.04 was the Bonnie++ benchmark, which tests the system's hard disk drive. The operations per second was relatively the same in the sequential create process while when performing random reads there was a drop off after 7.04 but it returned to surge ahead in 8.04 and 8.10. The Bonnie++ random read performance in Ubuntu 8.10 ended up being slightly ahead of Ubuntu 7.04. The random delete speed was also improved following Ubuntu 7.04. These disk performance improvements are likely attributed to the libata improvements in the recent Linux kernel releases.

Java: In the Java SciMark computational tests, the performance was slower in the Ubuntu 8.xx releases. This may be explained by IcedTea, but that shouldn't be since it's derived from OpenJDK and Sun's official Java source-code.

Major slowdowns after Ubuntu 7.04 "Feisty Fawn" in so many different tests certainly weren't what we had expected. This is our first time carrying out an Ubuntu comparison of recent releases on a large scale using a significant number of tests. It's likely the first time such a study has even been conducted. The tests that experienced performance losses initially we assumed were due to a regression with GCC but the tests extended beyond the ones built from source to include Java ones that use compiled byte-code and even the PHP-driven XML test.

A number of significant kernel changes had gone on between these Ubuntu Linux releases including the Completely Fair Scheduler, the SLUB allocator, tickless kernel support, etc. We had also repeated many of these tests to confirm we were not experiencing a performance fluke or other issue (even though the Phoronix Test Suite carries out each test in a completely automated and repeatable fashion) but nothing had changed. Ubuntu 7.04 was certainly the Feisty Fawn for performance, but based upon these results perhaps it would be better to call Ubuntu 7.10 the Gooey Gibbon, 8.04 the Hungover Heron, and 8.10 the Idling Ibex.

We will continue our testing process to see if we can find the underlying cause(s) of these performance problems along with testing out other hardware to see if this was an isolated hardware incident. You can test out the performance of your Linux desktop by installing the Phoronix Test Suite and running any of the numerous tests or suites that ship with this open-source software. We also intend on repeating these tests on other distributions to see if this is an Ubuntu-specific problem or perhaps there's a wider issue at hand.

There is quite a bit of testing going on within the free software community as it pertains to usability and compatibility, but not enough focus on performance. If this is indeed a widespread but unknown problem, hopefully it will be worked out in time for Ubuntu 9.04, especially with Ubuntu Linux appearing on more low-performance netbooks. Whatever the case may be, there's quite a bit of room for improving the performance on the Linux desktop. Your feedback can be shared in the Phoronix Forums and your test results uploaded to Phoronix Global.