Announcement

Collapse
No announcement yet.

GCC 4.6, LLVM/Clang 3.0, Open64 Benchmarks

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • GCC 4.6, LLVM/Clang 3.0, Open64 Benchmarks

    Phoronix: GCC 4.6, LLVM/Clang 3.0, Open64 Benchmarks

    For those interested by last week's articles on the AMD Bulldozer benchmarks of the GCC, Open64, and LLVM/Clang compilers and the FX-8150 compiler tuning, but curious to know how the new LLVM/Clang 3.0 and other compilers perform on other processors, here's some new test results. These tests are looking at GCC 4.6, LLVM/Clang 3.0-RC1, and AMD Open64 compilers on Intel Sandy Bridge and AMD Shanghai hardware.

    Phoronix, Linux Hardware Reviews, Linux hardware benchmarks, Linux server benchmarks, Linux benchmarking, Desktop Linux, Linux performance, Open Source graphics, Linux How To, Ubuntu benchmarks, Ubuntu hardware, Phoronix Test Suite

  • #2
    Ekopath

    What about that Ekopath compiler ? After all the fuss when it was open sourced I heard nothing about it. Maybe I missed something but still I'd like to see some results for that one too.

    Comment


    • #3
      SunStudio/Solaris studio comparison

      I would like to see comparison side-by-side with Solaris Studio (Formerly SunStudio).

      Comment


      • #4
        What I find most interesting is that Open64 performs inconsistently while consitently failing to compile on intel platforms.

        Still a big gap between llvm and gcc in one test too.

        Comment


        • #5
          I've often found the Phoronix test results very different from that of my own. The 7-zip test for instance, it yields very different results from what I get when I use the built in benchmark functionality so I figured I would do a real (as in non-synthetic) test using p7zip on my corei5 machine (nehalem):

          Code:
          GCC		-O1 -march=native		0m36.376s
          GCC		-O1 -march=native		0m36.369s		Profile guided optimization
          GCC4.7		-O1 -march=native		0m35.814s
          GCC4.7		-O1 -march=native		0m35.761s		Profile guided optimization
          Clang		-O1 -march=native		0m42.236s
          GCC		-O2 -march=native		0m35.734s
          GCC		-O2 -march=native		0m33.435s		Profile guided optimization
          GCC4.7		-O2 -march=native		0m35.663s
          GCC4.7		-O2 -march=native		0m33.831s		Profile guided optimization
          Clang		-O2 -march=native		0m39.658s
          GCC		-O3 -march=native		0m34.197s
          GCC		-O3 -march=native		0m33.428s		Profile guided optimization
          GCC4.7		-O3 -march=native		0m34.693s
          GCC4.7		-O3 -march=native		0m33.224s		Profile guided optimization
          Clang		-O3 -march=native		0m39.642s
          These are the results of compressing an .iso file (Arch linux netinstall to be exact) on a core i5, 64bit Arch Linux system using the following compilers:
          GCC 4.6.2-1, Clang 2.9-7, GCC 4.7 snapshot 20111105 (I built the LLVM 3.0 release aswell but it failed compiling stating it was unable to find libc++, I guess I'll wait for the full release to hit the Arch repos for that one)

          Anyway there wasn't much to compare against this time in Phoronix's test since for some reason there was no Clang/LLVM test and Open64 (4.2.4-2) wasn't able to build p7zip on my machine. But as has always been the case in my tests, GCC outperforms Clang/LLVM by a good margin.

          Comment


          • #6
            Oh and if anyone wants it, here's the (ugly, my bash-script-fu is very weak) bash script I threw together to do the test:

            Code:
            #!/bin/bash
            name=p7zip_
            version=9.20.1
            dir=$name$version
            results=../p7zip-results.txt
            makefile=makefile.linux_amd64
            arg="./bin/7za a -mx9 -m0=LZMA2:d64m -mmt dummy.7z archlinux-2011.08.19-netinstall-i686.iso"
            flags="-O3 -march=native"
            path64flags="-O3 -march=auto"
            
            # uncomment compilers to be tested
            gcc=1
            clang=1
            #gcc47=1
            #path64=1
            
            
            if [ ! -d $dir ]; then
            	echo "Downloading and extracting p7zip..."
            	wget -q http://downloads.sourceforge.net/project/p7zip/p7zip/${version}/p7zip_${version}_src_all.tar.bz2
            	bzip2 -q -d "$dir"_src_all.tar.bz2
            	tar xf "$dir"_src_all.tar
            fi
            
            cd $dir
            
            if [ ! -e archlinux-2011.08.19-netinstall-i686.iso ]; then
            	echo "Downloading iso..."
            	wget -q http://ftp.ds.hj.se/pub/os/linux/archlinux/iso/2011.08.19/archlinux-2011.08.19-netinstall-i686.iso
            fi
            
            
            if [ ! -z $gcc ]; then
            # GCC ----------------------------------------------------------------------------------------------------
            cp $makefile makefile.machine
            sed -i "s/OPTFLAGS=-O/OPTFLAGS=$flags/" makefile.machine
            make clean
            make -j5
            bench=`{ time ${arg}; } 2>&1 | grep real | awk '{print $2}'`
            echo -e "GCC\t\t$flags\t\t$bench" >> $results
            rm dummy.7z
            # pgo
            cp $makefile makefile.machine
            sed -i "s/OPTFLAGS=-O/OPTFLAGS=$flags -fprofile-generate/" makefile.machine
            make clean
            make -j5
            bench=`{ time ${arg}; } 2>&1 | grep real | awk '{print $2}'`
            rm dummy.7z
            cp $makefile makefile.machine
            sed -i "s/OPTFLAGS=-O/OPTFLAGS=$flags -fprofile-use -fprofile-correction/" makefile.machine
            make clean
            make -j5
            bench=`{ time ${arg}; } 2>&1 | grep real | awk '{print $2}'`
            echo -e "GCC\t\t$flags\t\t$bench\t\tProfile guided optimization" >> $results
            rm CPP/7zip/Bundles/Alone/*.gcda
            fi
            
            if [ ! -z $gcc47 ]; then
            # GCC 4.7 ----------------------------------------------------------------------------------------------------
            cp $makefile makefile.machine
            sed -i "s/OPTFLAGS=-O/OPTFLAGS=$flags/" makefile.machine
            sed -i 's/CXX=g++/CXX=g++-4.7/' makefile.machine
            sed -i 's/CC=gcc/CC=gcc-4.7/' makefile.machine
            make clean
            make -j5
            bench=`{ time ${arg}; } 2>&1 | grep real | awk '{print $2}'`
            echo -e "GCC4.7\t\t$flags\t\t$bench" >> $results
            rm dummy.7z
            # pgo
            cp $makefile makefile.machine
            sed -i "s/OPTFLAGS=-O/OPTFLAGS=$flags -fprofile-generate/" makefile.machine
            sed -i 's/CXX=g++/CXX=g++-4.7/' makefile.machine
            sed -i 's/CC=gcc/CC=gcc-4.7/' makefile.machine
            make clean
            make -j5
            bench=`{ time ${arg}; } 2>&1 | grep real | awk '{print $2}'`
            rm dummy.7z
            cp $makefile makefile.machine
            sed -i "s/OPTFLAGS=-O/OPTFLAGS=$flags -fprofile-use -fprofile-correction/" makefile.machine
            sed -i 's/CXX=g++/CXX=g++-4.7/' makefile.machine
            sed -i 's/CC=gcc/CC=gcc-4.7/' makefile.machine
            make clean
            make -j5
            bench=`{ time ${arg}; } 2>&1 | grep real | awk '{print $2}'`
            echo -e "GCC4.7\t\t$flags\t\t$bench\t\tProfile guided optimization" >> $results
            rm CPP/7zip/Bundles/Alone/*.gcda
            fi
            
            if [ ! -z $clang ]; then
            # CLANG/LLVM ---------------------------------------------------------------------------------------------
            cp $makefile makefile.machine
            sed -i "s/OPTFLAGS=-O/OPTFLAGS=$flags/" makefile.machine
            sed -i 's/CXX=g++/CXX=clang++/' makefile.machine
            sed -i 's/CC=gcc/CC=clang/' makefile.machine
            make clean
            make -j5
            bench=`{ time ${arg}; } 2>&1 | grep real | awk '{print $2}'`
            echo -e "Clang\t\t$flags\t\t$bench" >> $results
            rm dummy.7z
            fi
            
            if [ ! -z $path64 ]; then
            # PATH64 ---------------------------------------------------------------------------------------------
            cp $makefile makefile.machine
            sed -i "s/OPTFLAGS=-O/OPTFLAGS=$path64flags/" makefile.machine
            sed -i 's/CXX=g++/CXX=pathCC/' makefile.machine
            sed -i 's/CC=gcc/CC=pathcc/' makefile.machine
            make clean
            make -j5
            bench=`{ time ${arg}; } 2>&1 | grep real | awk '{print $2}'`
            echo -e "Path64\t\t$path64flags\t\t$bench" >> $results
            rm dummy.7z
            fi
            Put it in a directory and execute it, it will download p7zip and the iso used to compress the first time it's run and then perform the benchmarks and place the results in a textfile named p7zip-results.txt in the directory of the benchmark script. Just change the cflags in the script to those you want to test and also comment/uncomment the compilers you wish to test (note that open64 and path64 fails on p7zip on my machine). Also this assumes a 64bit system, if you want to benchmark on a 32-bit system you need to change the makefile variable to one of the 32-bit makefiles which accompany p7zip (weird buildsystem in p7zip, give me ./configure anyday!)

            Comment


            • #7
              Please use normal images in your posts!

              Comment


              • #8
                SmallPT benchmark is useless when comparing compilers

                SmallPT is not optimized at all in PTS, therefore when you compare to compilers you don't really compare the potential speed of their executables but something else which is not related to speed at all.

                CLANG/LLVM opts to generate slower code when in no optimization mode. It is slower than GCC when unoptimized but you can't learn anything about its code generation when optimizing for full speed.

                See for example the difference of optimized and unoptimized SmallPT. The difference is HUGE. The optimized version is more than 3 times faster.

                (My machine, Phenom II X6 @ 3.6 GHz)
                OpenBenchmarking.org, Phoronix Test Suite, Linux benchmarking, automated benchmarking, benchmarking results, benchmarking repository, open source benchmarking, benchmarking test profiles


                The smallpt test should be optimized in future versions of PTS

                Comment


                • #9
                  The SmallPT test isn't fair, as Clang don't support OpenMP, so it's not using multiple threads, which makes the test pointless...

                  And it's not made clear what optimization level was given to the compilers to build the tests? GCC's and LLVM's are different scales...

                  Comment


                  • #10
                    Most important question ... How did you manage to install Open64 5.0 on an Ubuntu 11.10 distribution ??

                    Comment

                    Working...
                    X