Announcement

Collapse
No announcement yet.

A Linux Compiler Deathmatch

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • A Linux Compiler Deathmatch

    Phoronix: A Linux Compiler Deathmatch

    Started by one of our readers more than a week ago was a compiler deathmatch for comparing the performance of GCC, LLVM Clang, PCC (the Portable C Compiler), TCC (Tiny C Compiler), and Intel's C Compiler under Arch Linux. This user did not stop there with compiling these different x86_64 code compilers, but he also went on to look at the compiler performance with different compiler flags, among other options. The results are definitely worth looking at and here are some more.

    http://www.phoronix.com/vr.php?view=15657

  • #2
    Is anyone benchmarking the quality of the output in terms of execution and code quality and stability ?

    More of a curiosity.

    Comment


    • #3
      It would be interesting to see how the Intel compiler alternative compares these Free ones.

      Comment


      • #4
        If you want to test the speed of the resulting binaries, PLEASE test also what the compiler is able to do and not only what it does in default configuration. Please include maximum optimization in the benchmarks.

        Comment


        • #5
          The Arch Linux benchmarks were a lot more interesting

          If this was a deathmatch, who won?

          Comment


          • #6
            If it's not a too big problem, could you please run a set of FORTRAN compiler tests, too? Please include ifort and gfortran for sure!

            The -O2 -march=native options seem to be the most interesting...

            Thank you!

            Comment


            • #7
              Please add a comparison on ARM too. And consider benchmarking at different optimization levels (eg. -Os, -O2 and -O3 for gcc)

              Comment


              • #8
                Michael, unless you tell the compiler to optimise, it won't optimise. These results are completely useless. It's like testing how fast different graphics cards render with the Mesa software rasteriser.

                The interesting results would be to see what effect different optimisation levels (-O2, -O3, -Os, plus compiler-specific features) have on the final executable. This will help individual developers and distro packagers to pick the best compiler options for specific programs. This would need about 5-6 different configurations per compiler and most of us don't have the time to do it at home, so we would be very grateful if you do a proper benchmark.

                Comment


                • #9
                  Completely useless test.
                  You can't compare compilers with their default settings as they have different optimizations by default.

                  So what should have been done is to compare the different optimization settings of the compilers.

                  Comment


                  • #10
                    "All compilers were tested in their "out of the box" configuration without specifying any extra flags."

                    This is just brain damaged. What a waste of time and resources. Seriously, this is from the same guy who *wrote* the Phoronix benchmarking suite?

                    I honestly can't believe this. Seriously. Could Michael please explain what's the point in this?

                    Comment


                    • #11
                      test

                      First of all. thanks Michael for supposed-to-be-interesting test. Now could you please repeat it with -O3 flags, so that new version actually make sense :-) .

                      Comment


                      • #12
                        i asked this once before but got no answer:

                        considering these results: how can anyone say a program has regressed or not if it could be the compiler just being sensitive for a certain program/procedure/call?
                        when numbers of 2 different versions of GCC differ by 10, 50, or sometimes by x00%

                        how can you write considerable and stable programs when the compiler seems not to be reliable?

                        Comment


                        • #13
                          Originally posted by jakubo View Post
                          i asked this once before but got no answer:

                          considering these results: how can anyone say a program has regressed or not if it could be the compiler just being sensitive for a certain program/procedure/call?
                          when numbers of 2 different versions of GCC differ by 10, 50, or sometimes by x00%

                          how can you write considerable and stable programs when the compiler seems not to be reliable?
                          Firstly let's get the language right. It's not about reliability. If any of these compilers are having reliability issues (i.e they are creating incorrect instructions) you should raise that with the various projects.

                          Now, if the question is of performance, the answer is you don't know. So if performance is of prime concern you should learn more about the compilers you are using and run tests

                          Comment


                          • #14
                            Originally posted by RealNC View Post
                            "All compilers were tested in their "out of the box" configuration without specifying any extra flags."

                            This is just brain damaged. What a waste of time and resources. Seriously, this is from the same guy who *wrote* the Phoronix benchmarking suite?

                            I honestly can't believe this. Seriously. Could Michael please explain what's the point in this?
                            99% of users use the default GCC configuration. Do you think anyone running Ubuntu packages is going to have -funroll-loops an -march=core2? Only loony Gentoo users like myself use anything but the default.

                            Even most users who compile a package or two themselves don't override the CFLAGS in the Makefile.

                            Comment


                            • #15
                              The way I understood it is that Michael has used no flags at all, not even plain -O2.

                              Comment

                              Working...
                              X