LLVM/Clang 3.2 Compiler Competing With GCC

[ArneBab]

Matrix Operations

It looks like LLVM/Clang mostly wins in SciMark, and these seem to be mostly matrix multiplications.

Sparse Matrix, Dense Matrix, Matrix Factorization, Jacobi *Matrix* Over-Relaxation.

Also there?s the Monte Carlo test, which might be implemented with Matrizes, too (I just wanted to check but could not find the sources).

So all these tests essentially evaluate the performance of matrix operations. And there I would ask: Which library?

When I installed scipy on our cluster (KIT HC3), it selected the best library among a number of libraries by actually compiling test code and doing runtime performance tests, because the libraries differ between platforms. Have a look at ATLAS ( http://math-atlas.sourceforge.net/ ) for some background.

The only other test where LLVM wins against GCC is Tachyon - and that likely uses matrix operations, too (because most 3D stuff can be represented by matrizes).

For all other tests LLVM loses. So the high number of Matrix tests skews the perception by placing a big focus on one feature.

[vertexSymphony]

These benchmarks are as useful as looking at the sky to know if it's gonna rain during the day ... they lack so much information !!
for example, I couldn't manage to find which switches were used on each compiler and test, if inline assembly was disabled or not, blabla.

Well, I'm glad that Clang "does good" in this kind of gross measure; But this benchmark it's not my coup of tea: I would prefer something which I can draw more meaningful conclusions from

[roodemol]

A) Just supporting LLVM doesn't make someone an "apple fanboy" so go and take your apparent GCC-fanboyism somewhere else.

There's also the license issue, some OSes are stuck with a relatively outdated version of GCC. They are about to move to clang.

B) "Hands down" is a BIG stretch. LLVM was extremely competitive in just about every test that doesn't require OpenMP, and once OpenMP IS sorted out I bet those benchmarks will show LLVM is about the same as GCC in those tests as well.

Some time ago Michael Larabel published a small article about LLVM's Polly: http://www.phoronix.com/scan.php?pag...tem&px=MTIwMzg. How advanced is this project, is it possible to perform these benchmarks with this polyhedral optimizer?

C) The benefits in LLVM aren't in speed. 1) Modularity-- something GCC cant accomplish without scrapping a large chunk of the codebase SPECIFICALLY because GNU WANTED a monolithic codebase so that companies couldn't pull in select portions of gcc and just use those. 2) Cleaner code base. 3) Competition finally amongst compilers finally. GCC has gone uncontested in the open source world for a very long time, its good to see a project finally giving it a run for its money and ensuring that they don't start to rest on their laurels

There's another open source competitor, Open64. I hope this will bring some movement into the compiler scene and probably one of the contestant will outperform GCC.

Regards,
Gilles

[roodemol]

A) Just supporting LLVM doesn't make someone an "apple fanboy" so go and take your apparent GCC-fanboyism somewhere else.

There's also the license issue, some OSes are stuck with a relatively outdated version of GCC. They are about to move to clang.

B) "Hands down" is a BIG stretch. LLVM was extremely competitive in just about every test that doesn't require OpenMP, and once OpenMP IS sorted out I bet those benchmarks will show LLVM is about the same as GCC in those tests as well.

Some time ago Michael Larabel published a small article about LLVM's Polly: http://www.phoronix.com/scan.php?pag...tem&px=MTIwMzg. How advanced is this project, is it possible to perform these benchmarks with this polyhedral optimizer?

C) The benefits in LLVM aren't in speed. 1) Modularity-- something GCC cant accomplish without scrapping a large chunk of the codebase SPECIFICALLY because GNU WANTED a monolithic codebase so that companies couldn't pull in select portions of gcc and just use those. 2) Cleaner code base. 3) Competition finally amongst compilers finally. GCC has gone uncontested in the open source world for a very long time, its good to see a project finally giving it a run for its money and ensuring that they don't start to rest on their laurels

There's another open source competitor, Open64. I hope this will bring some movement into the compiler scene and probably one of the contestant will outperform GCC.

Regards,
Gilles

[XorEaxEax]

Does anyone know how the two compare on ARM processors? This seems to be an area where the two compete head-to-head.

Yes this is an area which I'm also interested in, in x86 64 which I am benchmarking GCC still holds supreme in terms of code performance, however I have no idea of how these compilers compare on ARM.

And given that this is an extremely strong architecture outside of the desktop it would be very interesting to see which compiler toolchain has the best performance here.

[XorEaxEax]

GCC had major competition, what are you talking about?

Open source competition?

It's still the unpopular choice in a number of important fields where other compilers are dominant.

What are these in particular?

[XorEaxEax]

Any particular reason why in a few benhcmarks, LLVM is FAR FAR ahead of GCC ? It almost looks like GCC is broken is these benchmarks.

You can dismiss any comparison where there isn't a -On flag set as we have no idea what optimization level is used. If you don't set a compiler optimization flag, GCC will deafult to -O0, which is no optimization.

Also, there's no way of ensuring that -O0,-O1,-O2 on GCC and Clang/LLVM are in any way equivalent optimization levels between compilers as there is no 'law' on what optimizations to enable at these levels. As such they are pretty worthless for compiler vs compiler benchmarks, atleast if you exclude -O3.

-O3 on the other hand is the highest optimization level which favors speed above everything else, so if you are only going to benchmark _one_ optimization level then this is it. There are also other special optimizations like link time optimizations (available in both GCC and Clang/LLVM), profile guided optimization (in GCC but not yet in Clang/LLVM) but these are beyond the scope of these tests in my opinion as Michael already have a hard time with the standard -On levels from the looks of it.

So looking at these tests, only the following declare setting the -O3 flag (best performant code, which is what we are benchmarking here):

Timed HMMER Search
Timed MAFFT Alignment
Blake2
Himeno Benchmark
C-Ray
Smallpt (OpenMP so rather pointless to compare)
GraphicsMagic (OpenMP so rather pointless to compare, how about configuring with --disable-openmp ?)
Tachyon (-m32 ? doesn't this work with 64-bit? )

Then we have Lame MP3 which defaults to -O3, so that should be ok unless Michael has changed the settings, however afaik this relies on assembly optimizations aswell.

JohnTheRipper defaults to -O2 (ignores CFLAGS environment variable so you need to manually edit makefile) and uses lots of assembly optimizations which makes it a poor candidate for benchmarking. However afaik it only uses OpenMP if you set OMPFLAGS to -fopenmp (not default), so why Michael is enabling it for this test is beyond me as it only makes sense to have if both compilers being benchmarked supports it.

VP8 libvpx and x264 would be great candidates for testing, both default to -O3 and offer a great challenge for optimizing compilers, except of course that Michael compiles them with assembly optimizations _enabled_ which makes them pointless as the compilers are left with pretty much no performance critical code to optimize, hence the pretty much identical results in these tests all these years.

The SciMark tests have no declared settings, and while Michael states that he sets CFLAGS/CXXFLAGS to -O3 -march=native, not all packages use/honor these environment variables, as seems to be the case here (as is the case with JohnTheRipper). Until we know the settings actually used in these tests they are worthless.

With Phoronix up for sale and with Michael indicating that he wants to continue focusing on the benchmarking I really hope takes the time to actually learn/take the time to properly configure the test software he benchmarks so that the results have better meaning.

LLVM is doing pretty good. It has narrowed the differences quite a lot. But its the last 10% difference thats the hardest to overcome.

Yes there's obviously less low hanging fruit nowadays, obviously the lack of OpenMP is a big bottleneck and I don't understand why Michael persists in doing OpenMP based comparisons as of now at all as it's pointless until LLVM supports it. Just like it would be pointless to compare GCC + PGO against Clang/LLVM until it's supported in the latter.

[Sidicas]

Any particular reason why in a few benhcmarks, LLVM is FAR FAR ahead of GCC ? It almost looks like GCC is broken is these benchmarks.

LLVM is doing pretty good. It has narrowed the differences quite a lot. But its the last 10% difference thats the hardest to overcome.

I think you missed the part where in some charts, less is better.. While in other charts, more is better. LLVM is never far far ahead, it's far far behind in some tests. Supposedly due to lacking OpenMP support.

[smitty3268]

Any particular reason why in a few benhcmarks, LLVM is FAR FAR ahead of GCC ? It almost looks like GCC is broken is these benchmarks.

LLVM is doing pretty good. It has narrowed the differences quite a lot. But its the last 10% difference thats the hardest to overcome.

It most likely has to do with tunables - like at what point the compiler defaults to inlining methods versus using function calls, etc. Those are hard to get 100% right - there will always be some benchmarks that end up on the wrong side of the cutoff, but tweaking to speed them up may result in small losses across hundreds of other benchmarks and make it not worth it.

[mayankleoboy1]

Any particular reason why in a few benhcmarks, LLVM is FAR FAR ahead of GCC ? It almost looks like GCC is broken is these benchmarks.

LLVM is doing pretty good. It has narrowed the differences quite a lot. But its the last 10% difference thats the hardest to overcome.