Announcement

Collapse
No announcement yet.

CompilerDeathMatch: surprising results

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • #31
    nice work, staalmanen. Once you got your cflags tweaked for each compiler based on the input the other postes have provided you that produces the best optimizations then why not then do a final benchmark run?

    Also the unoptimized benchmarks (doing the compiles without any optimizations of special flags other than the architecture) are useful as a baseline.

    Comment


    • #32
      WOW, i never knew there are so any compilers for linux!
      I actually only knew gcc, icc and clang!

      But what strikes me more is the fact that for every task GCC does there seems to be a compiler out there that can do it faster and that can make a faster binary! In your benchmarks gcc never wins anywhere!

      However in that impressive chart i see one thing that just shocks me. APACHE static webpage serving! It seems, from your chart, that no mather how high you tune gcc, TCC and Open64 will nearly double the performance!

      Something odd is going on in your C-Ray benchmark.. every other benchmark has wildly varying scores, but C-Ray is nearly the same on every compiler.. That can't be right. (or is it..?)

      And if i look at it the right way it seems that using Open64 -o2 is beating ALL gcc benchmarks at -o2 and with some big differences as well! Now i wonder, why isn't everything compiled in Open64 since it obviously seems superior to GCC..

      Comment


      • #33
        Originally posted by markg85 View Post
        WOW, i never knew there are so any compilers for linux!
        I actually only knew gcc, icc and clang!

        But what strikes me more is the fact that for every task GCC does there seems to be a compiler out there that can do it faster and that can make a faster binary! In your benchmarks gcc never wins anywhere!

        However in that impressive chart i see one thing that just shocks me. APACHE static webpage serving! It seems, from your chart, that no mather how high you tune gcc, TCC and Open64 will nearly double the performance!

        Something odd is going on in your C-Ray benchmark.. every other benchmark has wildly varying scores, but C-Ray is nearly the same on every compiler.. That can't be right. (or is it..?)

        And if i look at it the right way it seems that using Open64 -o2 is beating ALL gcc benchmarks at -o2 and with some big differences as well! Now i wonder, why isn't everything compiled in Open64 since it obviously seems superior to GCC..

        As I said in my first post in this thread: "surprising results"

        Back then I ran the "out of the box" performance of the different compilers (similar to the recent Phoronix benchmark) and got completely surprised about the absolutely superior behavior of TCC being both FAST and producing very fast binaries.
        I try to run these things as unbiased as possible, but a part of me is somewhat pleased to see that there is serious competition to GCC out there since I think that diversity is a good thing.
        It would be very interesting to hear whether there are Gentoo (or other source-based distribution) users out there that are running complete Clang-compiled systems (via the LLL tool chain or similar) or perhaps even Open64-compiled systems.
        I have tried to google whether anyone has succeeded compiling linux with Open64 but did not find any evidence of that though...

        Comment


        • #34
          Originally posted by staalmannen View Post
          As I said in my first post in this thread: "surprising results"

          Back then I ran the "out of the box" performance of the different compilers (similar to the recent Phoronix benchmark) and got completely surprised about the absolutely superior behavior of TCC being both FAST and producing very fast binaries.
          I try to run these things as unbiased as possible, but a part of me is somewhat pleased to see that there is serious competition to GCC out there since I think that diversity is a good thing.
          It would be very interesting to hear whether there are Gentoo (or other source-based distribution) users out there that are running complete Clang-compiled systems (via the LLL tool chain or similar) or perhaps even Open64-compiled systems.
          I have tried to google whether anyone has succeeded compiling linux with Open64 but did not find any evidence of that though...
          Compiling the linux kernel "seems" to be possible with Open64 ... http://forums.amd.com/forum/messagev...hreadid=138368 (big LOL that i find that on the AMD forum the place i certainly didn't expect it)

          Comment


          • #35
            There's two odd things in this benchmarks..
            1- PCC don't complete 'timed apache compilation' but it appears in the 'static web page serving'.
            2- Clang seems to compile faster when optimization is turned on.

            Comment


            • #36
              Originally posted by tessio View Post
              There's two odd things in this benchmarks..
              1- PCC don't complete 'timed apache compilation' but it appears in the 'static web page serving'.
              2- Clang seems to compile faster when optimization is turned on.

              That is indeed strange. I do not really know how these tests are designed. I just started off with running the "compiler" suite and got data that later led to more data and so on...
              Perhaps the compilation time test is stricter on getting a "success" returned in order to approve the result, whereas the functional test of the resulting binary is more lenient when it comes to what the compilation returns?

              Comment


              • #37
                Cross post: Updated with -Os flags

                A new merge with the -Os results are hereby presented:

                http://global.phoronix-test-suite.co...-231-8340-1312

                Enjoy!

                Next up: Aggressive and compiler-specific settings (are there any tweaks that can be made to TCC?).

                Comment


                • #38
                  Originally posted by staalmannen View Post
                  A new merge with the -Os results are hereby presented:

                  http://global.phoronix-test-suite.co...-231-8340-1312

                  Enjoy!

                  Next up: Aggressive and compiler-specific settings (are there any tweaks that can be made to TCC?).
                  erm, wrong link....

                  Comment


                  • #39
                    nevermind, i found it. Here is the link: http://global.phoronix-test-suite.co...-231-8340-1312

                    Comment


                    • #40
                      SolarisStudio

                      I just downloaded SolarisStudio as well. Do people find it interesting that I also include this one in the set-up? This will delay the final closure of this experiment (and delay the start of 32-bit tests) further unfortunately, but would give it a wider scope.

                      Next up are compiler-specific flags none the less, and I will also try to rename the identifiers on all my experiments prior to the merge so that they get a nicer formatting (which definietly is needed now since there are so many data points).

                      Comment


                      • #41
                        Staalmannen thanks for this tests. It's interesting to see how -Os flag is perfoming pretty well on GCC and intel, in some cases even better than -O3 and -O2.

                        Comment


                        • #42
                          Originally posted by Jimbo View Post
                          Staalmannen thanks for this tests. It's interesting to see how -Os flag is perfoming pretty well on GCC and intel, in some cases even better than -O3 and -O2.
                          I am happy to hear that people find it interesting

                          I just uploaded compiler-specific flags for PCC, but I will not present a merge until I have added some of the most requested aggressive optimizations, primarily GCC and ICC.

                          I have just seen some openbenchmarking.org variants of my data, and I must say that the ability to group data (either per compiler or per optimization option) makes the data much easier to interpret. I hope we will be able to present such graphs soon

                          Ps. Thanks mtippett for your effort! Ds.

                          Cheers

                          Comment


                          • #43
                            Round of aggressive specific optimization flags...

                            The final round of the CompilerDeathMatch has started.

                            An intermediate update can be found here:
                            http://global.phoronix-test-suite.co...63-30048-18506

                            Changes:
                            - Optimization for PCC (suggested by Anders Magnusson, PCC mantainer)
                            - the suggested most aggressive optimizations for ICC and GCC
                            - SolarisStudio on default settings

                            The SolarisStudio is a late commer to the party, but will hopefully catch up soon.

                            Comment


                            • #44
                              Originally posted by staalmannen View Post
                              The final round of the CompilerDeathMatch has started.

                              An intermediate update can be found here:
                              http://global.phoronix-test-suite.co...63-30048-18506

                              Changes:
                              - Optimization for PCC (suggested by Anders Magnusson, PCC mantainer)
                              - the suggested most aggressive optimizations for ICC and GCC
                              - SolarisStudio on default settings

                              The SolarisStudio is a late commer to the party, but will hopefully catch up soon.
                              Suggestions/tips wanted for
                              - highly aggressive flags for Clang
                              - highly aggressive flags for SunCC
                              - A way to unclutter the data. To me, it looks like a lot of the graphs now fail to show up on the webpage. Perhaps I need to divide the merges per optimization group, although this would make the overall comparison less complete.

                              Comment


                              • #45
                                Originally posted by staalmannen View Post
                                Perhaps I need to divide the merges per optimization group...
                                It seems a good idea, in the current sate it's difficult to see detailed data.

                                Comment

                                Working...
                                X