Announcement

Collapse
No announcement yet.

CompilerDeathMatch 64bit Final results

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • #16
    I have a suggestion for aggregating results.

    For each benchmark, determine the average score across all compilers and flags. The for each compiler/flag, determine how much it deviates from the standard with (myScore-averageScore)/averageScore. For tests where less is better, multiply that by negative one. Then sum up these scores across tests and divide by total number of tests. (Time to compile and performance benchmarks should be done separately, I think).

    Basically, this would determine the average performance improvement (in percent) a particular compiler or set of flags would offer. A critic would be right to point out the limited utility of such an aggregation, but I still think it would be interesting to see.

    Comment


    • #17
      Originally posted by rexstuff View Post
      I have a suggestion for aggregating results.

      For each benchmark, determine the average score across all compilers and flags. The for each compiler/flag, determine how much it deviates from the standard with (myScore-averageScore)/averageScore. For tests where less is better, multiply that by negative one. Then sum up these scores across tests and divide by total number of tests. (Time to compile and performance benchmarks should be done separately, I think).

      Basically, this would determine the average performance improvement (in percent) a particular compiler or set of flags would offer. A critic would be right to point out the limited utility of such an aggregation, but I still think it would be interesting to see.
      I am not 100% sure that I understand exactly what you mean, but if I undestand you correctly, you want me to normalize each test and then put them together into some sort of "total performance" measure?

      The problem with this as far as I see is that different people will want to assign different weights to different tests - depending on their needs. In fact, the world is never nicely black-and-white but usually various shades of grey.

      Comment


      • #18
        Originally posted by rexstuff View Post
        I have a suggestion for aggregating results.

        For each benchmark, determine the average score across all compilers and flags. The for each compiler/flag, determine how much it deviates from the standard with (myScore-averageScore)/averageScore. For tests where less is better, multiply that by negative one. Then sum up these scores across tests and divide by total number of tests. (Time to compile and performance benchmarks should be done separately, I think).

        Basically, this would determine the average performance improvement (in percent) a particular compiler or set of flags would offer. A critic would be right to point out the limited utility of such an aggregation, but I still think it would be interesting to see.
        Well..

        With OpenBenchmarking.org (launching in just over a week), you can take the results and collapse them by either option class (-O2,-O1,-Os) or compiler (gcc, icc, etc). You can then select one compiler set (either aggregated or not) and normalize against that. You can also order by performance.

        We'll be there in just over a week, stay tuned.

        Comment


        • #19
          Fortran

          @staalmannen:

          Would you mind testing Fortran compilers as well? I think gfortran, g95, open64 and ifort would be the most interesting.

          Thanks for your time!

          Comment


          • #20
            Originally posted by HokTar View Post
            @staalmannen:

            Would you mind testing Fortran compilers as well? I think gfortran, g95, open64 and ifort would be the most interesting.

            Thanks for your time!
            The phoronix 3.0 compiler suite requires a fortran compiler too. Because of this I have packaged fort77 for the C-compilers that can not do fortran to do some comparisons.

            I am currently playing with unusual compilers only available for 32-bit.

            As we speak, I am running the KenCC (Plan9 C compiler, ancestor of the Golang compiler) for the pts/compilation suite
            CC=/opt/plan9/bin/pcc CXX=$CC (to avoid another c++ compiler to give results, looking at cfront to get results here... but I have not found a wrapper like fort77 for cfront)
            F77=fort77

            Extremely surprising is that the 3 first tests out of 5 in pts/compilation fails, whereas MPlayer and Linux compilation gives values... (if it were true it would be really cool since a future glendix could be self-hosting with a Plan9 userspace but I doubt it).


            One other problem I have right now: I can not register on openbenchmarking.org, the confirmation link only leads to the login page and login fails...

            Comment


            • #21
              Originally posted by staalmannen View Post
              As we speak, I am running the KenCC (Plan9 C compiler, ancestor of the Golang compiler) for the pts/compilation suite
              CC=/opt/plan9/bin/pcc CXX=$CC (to avoid another c++ compiler to give results, looking at cfront to get results here... but I have not found a wrapper like fort77 for cfront)
              F77=fort77
              These are the results for this initial thing
              http://openbenchmarking.org/result/1...IV-KENCCCOMP87

              Really surprised about getting a result from kernel compilation, and I doubt that it actually is a functional kernel. I need to check a few more things before being sure.

              I will do the "compiler" suite too for KenCC and then move on to Open Watcom, ACK (if I can make the build work) etc...

              some AUR links for people interested:

              AUcc
              http://aur.archlinux.org/packages.php?ID=26083
              OpenWatcom
              http://aur.archlinux.org/packages.php?ID=26963
              https://aur.archlinux.org/packages.php?ID=47685 (fortran)
              FTEQCC (???)
              http://aur.archlinux.org/packages.php?ID=29946
              nwcc
              http://aur.archlinux.org/packages.php?ID=23648
              Tendra
              http://aur.archlinux.org/packages.php?ID=18467
              KenCC
              https://aur.archlinux.org/packages.php?ID=47543
              ACK
              https://aur.archlinux.org/packages.php?ID=47568

              to-be-packaged:
              LCC
              http://sites.google.com/site/lccreta...iler/downloads

              Comment


              • #22
                I don't really understand you: what compilers did you use in the end? Why would you need a fortran wrapper to compile mplayer and the kernel?

                My idea was to get a big fortran code and compile it with gfortran and co. Am I missing something?

                Comment


                • #23
                  Originally posted by HokTar View Post
                  I don't really understand you: what compilers did you use in the end? Why would you need a fortran wrapper to compile mplayer and the kernel?

                  My idea was to get a big fortran code and compile it with gfortran and co. Am I missing something?
                  No the only reason Fortran was discussed was because when PTS got upgraded, the compiler suite benchmarks required not only a C compiler but also a Fortran compiler.
                  To make "fair" comparison between compilers that do compile fortran (like GCC, open64) and those that do not, those that do not get the fort77 wrapper.

                  For mplayer and the kernel, that should not have any influence.
                  Unfortunately all these odd compilers fail the vast majority of the tests. I have not had time to play much with it lately though. The stuff is out there however so if someone wants to have a go at it, it is there

                  I would especially be interested to know if someone is able to run PTS tests on openwatcom. This is a weird piece of compiler that I do not really know how to use properly

                  Comment

                  Working...
                  X