Benchmarking LLVM's Automatic Loop Vectorizer

Yesterday I wrote about an automatic loop vectorizer having been committed to LLVM this week. I've now carried out some benchmarks of this automatic loop vectorizer in conjunction with the Clang compiler to see the performance impact of this still experimental but promising feature.

As mentioned in Friday's article, the LLVM auto loop vectorizer is still deemed experimental and "far from being ready", thus is currently not enabled by default. This feature will premiere with LLVM 3.2 in December and can be enabled by passing the -mllvm -vectorize flags through the Clang C/C++ compiler.

While ultimately the automatic loop vectorizer should be a performance win, one of the Apple engineers working on this LLVM code wrote, "At the moment the vectorizer will vectorize anything it can, because we do not have a 'cost-model' to estimate the profitability of vectorization. Implementing a cost model is a high-priority for us, and until this is ready you should expect to see slowdowns on many loops. Another area which we need to improve is the memory dependence check. At the moment we have a very basic memory legality check which can be improved. Additionally, there are a number of cases where we generate poor vector code or suffer from a phase-rdering problem. Once we solve these problems we can continue to implement additional features."

Using the latest LLVM/Clang SVN code as of Friday, I carried out some early benchmarks of the automatic loop vectorizer when running some open-source Phoronix Test Suite / OpenBenchmarking.org tests. The only difference between test runs was setting the CFLAGS/CXXFLAGS for -mllvm -vectorize.

For some of the benchmarks, there was no measurable change in performance when the automatic loop vectorizer for LLVM was enabled.

The most common case, however, was actually a performance drop when the LLVM auto loop vectorizer was enabled. As mentioned, there isn't yet any cost-model for LLVM to determine when to vectorize a loop or not, plus other performance tuning of this newly-committed code is still needed.

Of the benchmarks run for this initial testing, the only test profile showing a positive improvement out of the automatic loop vectorizer was PostgreSQL.

More benchmarks and other system information/logs from the system during this initial LLVM auto loop vectorizer benchmarking can be found on OpenBenchmarking.org via the 1210264-RA-LLVMLOOPV66 result file.