LLVM Clang vs. GCC Compiler Benchmarks On FreeBSD 11.0

[phoronix]

Phoronix: LLVM Clang vs. GCC Compiler Benchmarks On FreeBSD 11.0

For those interested in C/C++ compiler performance, for some fun numbers to dive into this weekend are LLVM Clang vs. GCC benchmarks atop FreeBSD 11.0 RC1 AMD64 on an Intel Xeon Haswell system.

http://www.phoronix.com/vr.php?view=23476

[Brane215]

Hmmm. OK, compiler flags wer the same, but what were they ? And why no gcc-6.2.0 ? You announced it just a few days ago...

Also, it would be nice to have lto version improvements, if any, on some typical package as well as some known and lto-friendly package+lib combo...

[sdack]

Hmmm. OK, compiler flags wer the same, but what were they ? And why no gcc-6.2.0 ? You announced it just a few days ago...

Also, it would be nice to have lto version improvements, if any, on some typical package as well as some known and lto-friendly package+lib combo...

Seeing how llvm-3.9 is around the corner would it be more fair if gcc-6.2 gets matched up with llvm-3.9. Just my opinion. That it wasn't gcc-6.2 isn't really an issue. gcc-6.2 fixed mostly ICEs and a very few compile time issues compared to gcc-6.1 and so should at best only make a difference in the compile speed benchmarks, but I doubt it will change enough to make a difference. What surprised me was how well the gcc-6.1 generated code performed, all without the use of LTO. I had expected the older gcc's would perform about the same and better.

Would be interesting to know what the compiler flags were, but I am guessing these were all plain -O2. That's would I expect for a direct comparison without going into more detail and still to be meaningful and fair. A lot of code comes with -O2 as the default.

[microcode]

Maybe now we'll stop hearing all the bragging about compile times on Clang; given that they're basically on-par and have been for several releases of each compiler.

[pouar]

And FreeBSD brags about how much faster Clang is than GCC and blah blah blah. Maybe if they didn't use such an ancient version they would've gotten better performance.

[kenjo]

Maybe now we'll stop hearing all the bragging about compile times on Clang; given that they're basically on-par and have been for several releases of each compiler.

hu?? where was the compile time measurement?

[Brane215]

Seeing how llvm-3.9 is around the corner would it be more fair if gcc-6.2 gets matched up with llvm-3.9. Just my opinion. That it wasn't gcc-6.2 isn't really an issue. gcc-6.2 fixed mostly ICEs and a very few compile time issues compared to gcc-6.1 and so should at best only make a difference in the compile speed benchmarks, but I doubt it will change enough to make a difference. What surprised me was how well the gcc-6.1 generated code performed, all without the use of LTO. I had expected the older gcc's would perform about the same and better.

Would be interesting to know what the compiler flags were, but I am guessing these were all plain -O2. That's would I expect for a direct comparison without going into more detail and still to be meaningful and fair. A lot of code comes with -O2 as the default.

Fairness is not a concept I use when choosing as compiler. I wouldn't skip best available option just to be "fair" to a competitor. I suppose test was done for the benefit of the reader and his interest is probably information about available options.

Also it would be nice if one could get info about more aggressive and LTO options as probably not many packages get compiled with default -O2. New gccs are known to perform better with -O3 and especially LTO, so why not show this at least as an option ?

[sdack]

Fairness is not a concept I use when choosing as compiler. I wouldn't skip best available option just to be "fair" to a competitor. I suppose test was done for the benefit of the reader and his interest is probably information about available options.

Also it would be nice if one could get info about more aggressive and LTO options as probably not many packages get compiled with default -O2. New gccs are known to perform better with -O3 and especially LTO, so why not show this at least as an option ?

You mean you'd do what Volkswagen did... they optimized their motor control systems so far, that it detected emissions tests and gave "optimized" results - they ended up cheating. They, too, missed the point.

Of course will you have to be fair. Nobody will take benchmarks serious where one compiler uses -O2 and another gets to use -O3.

The problem with the options is that when you start using all available options then you are doing it wrong, because realistically would you select options for each compiler and every single benchmark individually. Only to find the best options for one compiler with a particular benchmark on a given hardware is time consuming, the result will only be meaningful to each particular case, but not be of much use for a general comparison.

The point of the benchmarks is to give the compilers something to do and to see how these compare with default settings, but not to find out how well one can optimize each benchmark. That's a completely different task.

Compilers are also meant to be smart enough to make most optimization decisions for you and not to demand of you to do these for the compiler. Another reason not to delve further into the massive amount of options (and parameters) that are available.

Maybe an article on LTO itself would be nice. Perhaps in combination with a single application (i.e. ffmpeg or firefox) on 2 or 3 different CPUs and gcc versus clang.

[carewolf]

Hmmm. OK, compiler flags wer the same, but what were they ? And why no gcc-6.2.0 ? You announced it just a few days ago...

Also, it would be nice to have lto version improvements, if any, on some typical package as well as some known and lto-friendly package+lib combo...

6.2 is just a patch release, it is not likely to produce better results. Mostly patch releases just fixes bugs on obscure code or obscure platforms.

[carewolf]

And FreeBSD brags about how much faster Clang is than GCC and blah blah blah. Maybe if they didn't use such an ancient version they would've gotten better performance.

I am actually surprised they shipped 4.6, that means they didn't stop at the last GPLv2 release 4.2, but smartly continued with the GPLv3 gcc releases, just several years behind.