Announcement

Collapse
No announcement yet.

Bringing Up Advancements In Compiler Testing

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Bringing Up Advancements In Compiler Testing

    Phoronix: Bringing Up Advancements In Compiler Testing

    For those interested in compiler testing and tuning performance, there's some notable advancements to share that will benefit the community...

    http://www.phoronix.com/vr.php?view=MTA2NzU

  • #2
    Thanks for the improvements, it looks great! Looks like 3.8 will be a great release

    Comment


    • #3
      Yes! I do complain alot about the Phoronix tests because they often use totally pointless optimization options, like -O0, -O1, targeting other architectures than the benchmarks are being run on (-march) which actually ends up benefiting compilers doing a poorer job optimizing code for a particular architecture. Stating that the upstream source tarballs comes with these options still doesn't make them relevant since no packager will distribute these applications compiled with -O0 or -O1. Primarily the packages in use on these systems are compiled with -O2 or -O3 and thus these conservative 'upstream' options are irrelevant as they are not what is in use on people's machines.

      Now I HOPE this means that we will see the official benchmarks compiled with these options (-O2, -O3) but if not then atleast it's important to have the compiler options displayed in each benchmark so that people will be able to take them with a grain of salt when they see the tests are being done on -O0, -O1 levels (in other words, worthless). The compiler options used should ALWAYS be displayed, else the results can't be accurately judged.

      Here is an example of a benchmarking test where you have all the necessary data provided to you to not only accurately compare the results but also be able to validate them for yourself:

      http://willus.com/ccomp_benchmark2.shtml

      Comment


      • #4
        Very nice

        Hey, this looks very useful.

        A suggestion about what else could be done:

        It would be nice if you could detect which options a compiler is ignoring - for example, some gcc flags are ignored when you use clang instead and I don't think your logging would show any difference.

        I'm not sure there is a simple way of determining that, though - so this may be hard to accomplish.

        Comment


        • #5
          Originally posted by smitty3268 View Post
          A suggestion about what else could be done:

          It would be nice if you could detect which options a compiler is ignoring - for example, some gcc flags are ignored when you use clang instead and I don't think your logging would show any difference.

          I'm not sure there is a simple way of determining that, though - so this may be hard to accomplish.
          Right, I'm not sure that there is any way with GCC or any other major compiler to dump what options are actually being utilized in generating a particular binary.
          Michael Larabel
          http://www.michaellarabel.com/

          Comment


          • #6
            Originally posted by smitty3268 View Post
            Hey, this looks very useful.
            It would be nice if you could detect which options a compiler is ignoring - for example, some gcc flags are ignored when you use clang instead and I don't think your logging would show any difference.

            I'm not sure there is a simple way of determining that, though - so this may be hard to accomplish.
            Well this is the reason I find -O3 the most 'fair' of optimizations to compare with since at this level all compilers are favoring code performance, although -O2 is also targeting performance it's more conservative in terms of code size. Obviously the best is to show both -O2, -O3, but again if you only choose one then I say -O3.

            Also which gcc flags are you referring too? I know that clang doesn't currently support gcc's -Ofast flag which expands to -O3 -ffast-math (which are both supported on clang), also previously it didn't support -march=native but that works in the latest version. Other than that clang has -O4 which turns on link time optimization which would be the same as -O3 -flto -fwhole-program on gcc (clang supports -flto -fwhole-program afaik so that should cause no problem).

            Still I think link-time optimization and profile-guided optimization lies outside the realm of these tests (profile-guided optimization isn't even available in clang/llvm) and I would be happy just to see the tests done with sane standard optimizations like -O2 and -O3, enough with the crazy -O0 -O1, -march=k8 settings which have plagued previous benchmark tests. -ffast-math could be interesting though in floating point intensive applications, here's hoping these benchmarks can finally live up to their potential.

            Comment


            • #7
              Originally posted by Michael View Post
              Right, I'm not sure that there is any way with GCC or any other major compiler to dump what options are actually being utilized in generating a particular binary.
              Well the options used in a binary is governed by the -OX flags or by explicitly enabling them (-f<optimization>). You can look at http://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html to see exactly which optimization options are enabled on each -OX level.

              Also looking at the screenshots of the tests I think they look good with one caveat in the last shot (stream copy) where the compiler options aren't shown, why not? Just showing them at the bottom would make it perfect.

              Comment


              • #8
                Originally posted by XorEaxEax View Post
                Well this is the reason I find -O3 the most 'fair' of optimizations to compare with since at this level all compilers are favoring code performance, although -O2 is also targeting performance it's more conservative in terms of code size. Obviously the best is to show both -O2, -O3, but again if you only choose one then I say -O3.

                Also which gcc flags are you referring too? I know that clang doesn't currently support gcc's -Ofast flag which expands to -O3 -ffast-math (which are both supported on clang), also previously it didn't support -march=native but that works in the latest version. Other than that clang has -O4 which turns on link time optimization which would be the same as -O3 -flto -fwhole-program on gcc (clang supports -flto -fwhole-program afaik so that should cause no problem).

                Still I think link-time optimization and profile-guided optimization lies outside the realm of these tests (profile-guided optimization isn't even available in clang/llvm) and I would be happy just to see the tests done with sane standard optimizations like -O2 and -O3, enough with the crazy -O0 -O1, -march=k8 settings which have plagued previous benchmark tests. -ffast-math could be interesting though in floating point intensive applications, here's hoping these benchmarks can finally live up to their potential.
                I wasn't really thinking of any optimization in particular, and they would vary from compiler to compiler. One that Mesa uses, just for an example, is -fno-builtin-memcmp. I'm not sure that really makes a difference in the benchmarks, since it's mostly meant to work around a performance problem in gcc itself, but it's noteworthy that there is a difference there between compilers. Something like march=native would be more important - even if clang has added this option, other compilers may not have.

                Comment


                • #9
                  Yes this is a problem when comparing compilers as there are no 'rules' as to which optimizations compilers will enable at for example -O2, so one compiler may use more optimizations on this level than another which would lead one to believe that compiler 'A' is better, and then when you compile using -O3 compiler 'B' wins instead.

                  Again this is why I think -O3 is the best option if you only test one optimization level since it's 'supposed' to generate the fastest code across the compilers, but even better still is comparing the compilers at both -O2 and -O3. In particular I recall this was a problem for the open64 compiler which unlike gcc, clang, and icc had a very big performance difference between -O2 and -O3 and when compared against these other compilers at -O2 it looked really bad while at -O3 it was really competitive.

                  Comment


                  • #10
                    yet again i ask for a full x264 supplied "checkasm --bench" , as that would be a very good option to add to the Phoronix standard tests too for both old and new platforms

                    Comment


                    • #11
                      Originally posted by popper View Post
                      yet again i ask for a full x264 supplied "checkasm --bench" , as that would be a very good option to add to the Phoronix standard tests too for both old and new platforms
                      Patches / new test profiles are always welcome too.
                      Michael Larabel
                      http://www.michaellarabel.com/

                      Comment


                      • #12
                        Originally posted by Michael View Post
                        Patches / new test profiles are always welcome too.
                        great, lets hope some PHP dev writes some one day, and you get around to writing a simple working and tested howto write test and finaly integrate a working PHP test profile for your data collecting product where people can actually find it so people can help you if they are so inclined to for free.
                        Last edited by popper; 03-08-2012, 04:47 PM.

                        Comment


                        • #13
                          Originally posted by popper View Post
                          great, lets hope some PHP dev writes some one day, and you get around to writing a simple working and tested howto write test and finaly integrate a working PHP test profile for your data collecting product where people can actually find it so people can help you if they are so inclined to for free.
                          Huh? There's already many independent people creating their own test profiles and test suites.... There is some documentation as well but looking at other existing test profiles is often the easiest and simplest way. Test profiles don't even touch PHP at all but is just a set of bash scripts and XML files.
                          Michael Larabel
                          http://www.michaellarabel.com/

                          Comment


                          • #14
                            Very nice feature, be sure to highlight the difference between the same compiler but with different version (e.g. gcc-4.4 vs gcc-4.6)

                            Thank you very much

                            Comment


                            • #15
                              Originally posted by enrico.tagliavini View Post
                              Very nice feature, be sure to highlight the difference between the same compiler but with different version (e.g. gcc-4.4 vs gcc-4.6)
                              That's long been reflected from the system information table.
                              Michael Larabel
                              http://www.michaellarabel.com/

                              Comment

                              Working...
                              X