Compiler Benchmarks Of GCC, LLVM-GCC, DragonEgg, Clang

LLVM 2.8 was released last month with the Clang compiler having feature-complete C++ support, enhancements to the DragonEgg GCC plug-in, a near feature-complete alternative to libstdc++, a drop-in system assembler, ARM code-generation improvements, and many other changes. With there being great interest in the Low-Level Virtual Machine, we have conducted a large LLVM-focused compiler comparison at Phoronix of GCC with versions 4.2.1 through 4.6-20101030, GCC 4.5.1 using the DragonEgg 2.8 plug-in, LLVM-GCC with LLVM 2.8 and GCC 4.2, and lastly with Clang on LLVM 2.8.

Beginning with our GNU Compiler Collection benchmarking, we benchmarked every recent major release: GCC 4.2.1, 4.3.0, 4.4.0, 4.5.0, and then we used the GCC 4.6.0 2010-10-30 development snapshot. GCC 4.5 was released in April and due to the amount of work in recent months, we found it worthwhile to include the GCC 4.6 development snapshot to see where we are heading into 2011 when this next major compiler release should arrive. Going back to GCC 4.2.1 was done since that is the version for which LLVM-GCC is based and the GCC 4.2 was the last series to be released under the GNU GPLv2 (newer versions are on the GPLv3). All of these releases were built from source with C and C++ support, which are the languages at the focus of this article. When it came to building all of these compilers for GCC and the LLVM-based test candidates, Ubuntu Maverick's GCC 4.4 system compiler was used.

LLVM 2.8 was built in its --enable-optimized with --disable-assertions flags from source. The DragonEgg 2.8 benchmarking was built per the project's instructions with LLVM 2.8 and a patched GCC 4.5.1. The LLVM-GCC setup was using the 2.8 compiler front-end against GCC 4.2.1. The operating system stack was Ubuntu 10.10 (x86_64) with the Linux 2.6.35 kernel, X.Org Server 1.9.0, and an EXT4 file-system.

While GCC needs not any introduction, LLVM-GCC is the LLVM C front-end (and LLVM-G++ is the C++ front-end) that uses a modified version of GCC 4.2 while plugging in the code generator and optimizers from the Low-Level Virtual machine. DragonEgg is similar to LLVM-GCC in that it replaces GCC's optimizers and code generators with those from LLVM, but rather than directly mangling with the GCC code-base itself, DragonEgg is a GCC plug-in that works with the plug-in architecture of GCC 4.5 and later. DragonEgg is designed to work with upstream GCC that is capable of handling plug-ins, but DragonEgg is not quite there yet as it currently does require a small (1.5KB) patch against the GCC 4.5 code-base that slightly changes around four functions. DragonEgg is also limited at this point to supporting i686 and x86_64 architectures and currently has troubles with some of GCC's supported languages such as Java and Objective-C. DragonEgg itself is a shared object that can then be dynamically loaded by GCC to provide LLVM's generators and optimizers.

Clang is the native front-end compiler for the Low-Level Virtual Machine with a focus on supporting the C, C++, Objective-C, and Objective-C++ languages. Clang does not leverage any part of GCC, and with LLVM/Clang 2.8, the C and C++ support is considered feature-complete. Clang is designed to compile programs faster and with less memory, is compatible with many GCC commands, is LLVM BSD licensed, provides a rather nice static analyzer, and is modularized. Clang is considered production quality on i686, x86_64, and ARM architecture targets. Where LLVM-GCC and DragonEgg have the advantage over Clang is being able to support a greater number of programming languages (those supported by GCC) such as Fortran and ADA where as Clang provides no level of support. There are also programs not yet able to be compiled by Clang, but can be built with GCC when using the optimizer/generator portions of LLVM. Some of the milestones that Clang has hit include replacing GCC in the FreeBSD base, Clang is now self-hosting, this LLVM compiler can compile the Boost libraries, and most recently Clang was able to build a working Linux kernel.

To ensure an accurate look at things, all of these compiler tests were carried out on three different systems with distinguishable hardware differences. One of the test systems was a Lenovo ThinkPad T61 notebook with an Intel Core 2 Duo "Penryn" processor, the other is our latest high-end system that has an Intel Core i7 970 "Gulftown" article with 12 threads and has been used for a few interesting articles already, like looking at the threaded LLVMpipe performance and with benchmarking five years worth of Linux kernels, and lastly we used one of our Opteron setups that had a single Shanghai Quad-Core CPU.

The Penryn-based ThinkPad T61 had an Intel Core 2 Duo T9300 (2.50GHz; dual-core), 4GB of system memory, 100GB Hitachi HTS72201 Serial ATA HDD, and a NVIDIA Quadro NVS 140M GPU. The Gulftown-based system had the Core i7 970 six-core CPU + Hyper Threading running at 3.20GHz (and 3.46GHz turbo frequency), an ASRock X58 SuperComputer motherboard, 3GB of system memory, a 60GB OCZ Vertex 2 SSD, and an ATI Radeon HD 5770 graphics card. Lastly, on the AMD side was the Shanghai system carrying one AMD Opteron 2384 Quad-Core CPU clocked at 2.70GHz, a Tyan S2927 motherboard with NVIDIA MCP55 chipset, 4GB of ECC Registered DDR2 memory, an OCZ Agility EX 64GB SSD, and an ATI Radeon HD 4870 graphics card.

All of this testing was done via the Phoronix Test Suite. With this being another round of testing powered by the latest Phoronix Test Suite 3.0 "Iveland" development version, there are more advanced graphs displayed in the article, etc. Not only that, but this compiler testing was also initiated by the next-generation version of Phoromatic presently under development powered by Iveland and will be integrated with the OpenBenchmarking.org infrastructure along with bringing other features for those using this remote test management primarily designed for large enterprise test farms. The first-generation Phoromatic allowed benchmarking systems across the world, such as running automated tests in America while controlling them on the Capri Island in Italy. The next-generation Phoromatic builds upon this with mobile optimizations and other improvements; and because of this these GCC and LLVM compiler performance benchmarks were able to come into my Apple iPhone while spending less time on my birthday weekend in front of a computer and more time surfing.

The benchmarks ran with the eight different compilers included Apache, Gcrypt, OpenSSL, John The Ripper, BYTE, C-Ray, Crafty, Himeno, MAFFT, HMMer, 7-Zip, LAME MP3, x264, GraphicsMagick, timed Apache compilation, and timed ImageMagick compilation.