Announcement

Collapse
No announcement yet.

PathScale Open-Sources The EKOPath 4 Compiler Suite

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Originally posted by codestr0m View Post
    Check the path64-dev mailing list. I think the patch was like 1-5 lines with .org blah blah in the subject. Honestly don't spend time on this unless

    1) You're ready to dig into the boot issues you'll encounter afterwards
    2) You're ready to file good bug reports so we can work with you in getting them fixed

    Please don't take this as discouraging, but it's a lot of work before we can support building the kernel in a high quality manor.
    thanks for the information. this is helpful. i'll have a look. Yes, I kind of suspected there might be some boot issues - Intel's C compiler was the exact same way. but with some effort i was able to get it to work (after a few frustrating failed attempts...lol)...and seeking help through the Intel software developers portal / forums.... (this was 2-3 years ago)

    I didn't take your comments as discouraging at all. you are just being honest and realistic, and i appreciate that.. it would be silly to have the expectation of having no issues at all, when substituting GCC with any other compiler for compiling the linux kernel...

    When i get some free time, i will probably give it a shot. Maybe i will use a VM, and I can probably provide a decent bug report. but i will do some more investigating first. right now, im just happy to be able to use pathCC. I've already re-compiled a few libraries and applications with it, without too many hassles.

    thanks codestrom,

    cheerz
    Last edited by ninez; 06-15-2011, 02:01 AM.

    Comment


    • For those who wants to see the Open64 commit log from pathscale
      (http://svn.open64.net/log.php?repname=Open64&path=%2F&)

      Rev Log message Author Age Path
      66 Bootstrap the PathScale branch from the pro64 sources. marcel 2035d 06h /
      71 Import of PathScale, version 1.0

      The PathScale sources have been pre-molded in the following way:
      1. The CVS keywords have been normalized for this repository and do
      not generate any diffs. For filenames this means that they have
      osprey1.0 and not kpro64 as the first level directory even though
      the file lives under kpro64.
      2. The CVS Log keyword in kpro64/arith/edition.c has been removed
      and the delta caused by it due to Fred Chow's initial import has
      been reverted.
      3. The cygnus_20030612 directory has been renamed to cygnus. This
      to reduce diffs with the Pro64 compiler, which had cygnus_20000828
      and that got renamed to cygnus before importing. Future PathScale
      releases have renamed this directory to cygnus as well.

      Diff-reduction has been applied in the following way:
      1. Source files that only contained copyright assertions but no other
      (non-whitespace) changes have been reverted.

      This commit does not remove files that are part of Pro64 but are not
      part of PathScale 1.0. There are files we do not want to lose, so this
      takes some additional effort. Also, no changes have been made yet to
      account for the cygnus_20030612->cygnus name change. This will all be
      resolved in follow-up commits. marcel 2034d 09h /
      75 Post-import fixups of the PathScale bits to account for the renaming of
      cygnus_20030612 to (just) cygnus. marcel 2034d 07h /
      77 Remove files not part of PathScale 1.0. Files under the following
      directories were kept:
      kpro64/common/com/ia64
      kpro64/init
      kpro64/kgnu_common/libio
      kpro64/kgnu_common/libstdc++
      kpro64/targia64
      kpro64/targcygnus_ia32_ia64
      kpro64/targia32_ia64_nodebug

      The libio and libstdc++ directories under the kgnu_common directory
      are kept out of conservatism. I'm not sure at this time they should
      be nuked and since they're not in the way, I can defer removing them
      until I do know. Though, they really shouldn't be there... marcel 2034d 05h /
      78 Branched off the /projects/pathscale branch after the 1.0 import.
      Used for development based on the 1.0 sources. marcel 2030d 09h /
      83 Import of PathScale version 1.3. marcel 2023d 07h /
      84 Assert PathScale copyright for changes made in version 1.0 for which
      no copyright was asserted then, but for which copyright is asserted
      now. marcel 2023d 07h /
      85 Bootstrap the PathScale 1.3 sandbox. marcel 2023d 06h /
      86 Import of PathScale, version 2.2.1 marcel 2022d 09h /
      87 The GNU linker testsuite was removed from PathScale 1.3. marcel 2022d 08h /
      88 kgnu_common was removed from PathScale 2.2.1
      libelf and libiberty were moved a directory level up.
      We previously kept libstc++ and libio, so we lost those
      with this commit. marcel 2022d 08h /
      89 Remove files not part of PathScale 2.2.1. Some probably weren't
      even part of PathScale 1.3, but I forgot too clean that up before
      importing 2.2.1. marcel 2022d 08h /
      90 Branch off the PathScale 2.2.1 import. marcel 2022d 08h /
      135 Import PathScale, version 2.2.1 (take 2). marcel 2010d 05h /
      136 Remove some Pro64 left-overs that we should not need, especially
      since pathScale doesn't need it. marcel 2009d 12h /

      Comment


      • Originally posted by ninez View Post
        thanks for the information. this is helpful. i'll have a look. Yes, I kind of suspected there might be some boot issues - Intel's C compiler was the exact same way. but with some effort i was able to get it to work (after a few frustrating failed attempts...lol)...and seeking help through the Intel software developers portal / forums.... (this was 2-3 years ago)

        I didn't take your comments as discouraging at all. you are just being honest and realistic, and i appreciate that.. it would be silly to have the expectation of having no issues at all, when substituting GCC with any other compiler for compiling the linux kernel...

        When i get some free time, i will probably give it a shot. Maybe i will use a VM, and I can probably provide a decent bug report. but i will do some more investigating first. right now, im just happy to be able to use pathCC. I've already re-compiled a few libraries and applications with it, without too many hassles.

        thanks codestrom,

        cheerz
        I wonder if the work done on the Clang-compiled Linux kernel could be of benefit also to a Path64 - compiled kernel?

        https://github.com/lll-project

        Comment


        • Originally posted by staalmannen View Post
          I wonder if the work done on the Clang-compiled Linux kernel could be of benefit also to a Path64 - compiled kernel?

          https://github.com/lll-project
          if i remember correctly, for that project doesn't clang itself have to be patched in order to make it work???

          Also, from my own experiences compiling the linux kernel with different compilers, usually the set of problems from compiler to compiler are very different...but i'm no expert - just some experience here and there...

          i actually had that kernel installed for fun (easy install on Archlinux using AUR). but it wasn't as good as the kernel i usually use, so i removed it.

          Comment


          • Originally posted by ninez View Post
            if i remember correctly, for that project doesn't clang itself have to be patched in order to make it work???

            Also, from my own experiences compiling the linux kernel with different compilers, usually the set of problems from compiler to compiler are very different...but i'm no expert - just some experience here and there...

            i actually had that kernel installed for fun (easy install on Archlinux using AUR). but it wasn't as good as the kernel i usually use, so i removed it.
            Good to know I thought that one problem with compiling Linux with alternative compilers was that the code assumed GCC-specific behaviour. If that was the case, I thought it was logical that the set of problems should be about the same for all other compilers (enforcing true standard C rather than de-facto standard GCC-type C). On the other hand, if each other compiler solves the compiling Linux problem by "behaving like GCC", I guess different compilers might end up with different sub-sets of that behaviour implemented...

            Yay for more diversity
            Now Linux can be compiled with ICC (although useless for redistribution due to license terms), Clang and Path64. Reportedly, it was once possible to compile Linux with TCC too (I have not tried myself if this is still possible). I suppose it is just a manner of time before PCC can do the same (it compiles FreeBSD already).

            Comment


            • Originally posted by staalmannen View Post
              Good to know I thought that one problem with compiling Linux with alternative compilers was that the code assumed GCC-specific behaviour. If that was the case, I thought it was logical that the set of problems should be about the same for all other compilers (enforcing true standard C rather than de-facto standard GCC-type C). On the other hand, if each other compiler solves the compiling Linux problem by "behaving like GCC", I guess different compilers might end up with different sub-sets of that behaviour implemented...

              Yay for more diversity
              Now Linux can be compiled with ICC (although useless for redistribution due to license terms), Clang and Path64. Reportedly, it was once possible to compile Linux with TCC too (I have not tried myself if this is still possible). I suppose it is just a manner of time before PCC can do the same (it compiles FreeBSD already).
              Last time i checked LinuxDNA is pretty much a dead-project. (these were the guys patching the kernel for the intel compiler). I belong to the google group, and the last releqase i saw was 2.6.36. and there isn't much activity going on. (at all really). the one guy who was giving help to the odd person using x86 (32bit) - decided one day to start demanding that people pay him the value of the Intel XE compiler suite (like $500USD at the time) for his help or patch. (it was a lot of work for him to patch every kernel release, but still not worth me shelling out more money than the laptop i was using at the time was worth...lol). that was sort of the beginning of the end for me using Intel's compiler...lol. (with the licensing issues being a problem also)..

              I think ekopath is better anyway...so far i have found it to be more to my liking and not a huge adjustment to use. (less than ICC anyway - for most apps/libs i have tried so far). But i haven't tried the kernel or gimp yet - so we will see what happens.

              ..about different compilers - as someone else pointed out (maybe codestrom?) compilers are inherently very complex creatures. they also can be very different internally - in how they actually work - so i tend to think that they each handle compatibility with GCC in different ways, that is likely not necessarily that transferable...this is my feeling anyway. but again, i am no expert
              Last edited by ninez; 06-16-2011, 01:53 AM.

              Comment


              • Originally posted by staalmannen View Post
                I wonder if the work done on the Clang-compiled Linux kernel could be of benefit also to a Path64 - compiled kernel?

                https://github.com/lll-project
                Coincidently - Bryce who afaik did most of the work for getting LLVM to build/boot the linux kernel also helped us. We had pretty good progress before (thank Gautam), but he did the last mile. Other than experience I don't think anything else between the projects could be shared.

                Comment


                • Originally posted by ninez View Post
                  Last time i checked LinuxDNA is pretty much a dead-project. (these were the guys patching the kernel for the intel compiler). I belong to the google group, and the last releqase i saw was 2.6.36. and there isn't much activity going on. (at all really). the one guy who was giving help to the odd person using x86 (32bit) - decided one day to start demanding that people pay him the value of the Intel XE compiler suite (like $500USD at the time) for his help or patch. (it was a lot of work for him to patch every kernel release, but still not worth me shelling out more money than the laptop i was using at the time was worth...lol). that was sort of the beginning of the end for me using Intel's compiler...lol. (with the licensing issues being a problem also)..

                  I think ekopath is better anyway...so far i have found it to be more to my liking and not a huge adjustment to use. (less than ICC anyway - for most apps/libs i have tried so far). But i haven't tried the kernel or gimp yet - so we will see what happens.

                  ..about different compilers - as someone else pointed out (maybe codestrom?) compilers are inherently very complex creatures. they also can be very different internally - in how they actually work - so i tend to think that they each handle compatibility with GCC in different ways, that is likely not necessarily that transferable...this is my feeling anyway. but again, i am no expert
                  If you think of a compiler as a front-end and a backend then most of the source level compatibility happens at the front-end. In the case of GCC it has also adds a bunch of extensions for developers to use. The path64 front-end is gcc-4.2 which makes the parsing/semantic analysis easy, but the problem comes for handling those extensions on the backend. In some cases it's trivial and we just haven't gotten around to it. (Maybe they are rarely used or worse a stupid idea :P) For the most part we handle the most common ones and as a result it's easier for people to migrate code from GCC to Path64/EKOPath. The linux kernel poses a couple additional challenges

                  1) Heavy GNU extension use (We're missing some here)
                  2) Heavy inline asm
                  3) Debugging the bootup process (It's not easy to debug why the SMP built kernel doesn't boot)

                  Another example is glibc. In this case making our GCC front-end emit IR that is correct for the backend was a major pita. With ENZO we've made *huge* changes around this area and now a lot easier. (GCC/GNU code is gone entirely and why you may see ENZO building an small linux distribution from the ground up before EKOPath)

                  Comment


                  • Am I doing something wrong?

                    Hi!

                    I'm evaluating the speed of the pathscale compiler. So I downloaded the nightly version from their homepage and ran some programs I compiled using this very compiler and compared the speed against GCC's version of the compiled code. In most of the cases GCC was faster. This was my own private source code, heavily relying on the Mersenne-Twister-Algorithm. I tested Matsumoto's version (dSFMT) and Boost's version.

                    I took the Boost implementation of Mersenne-Twister as reference, as I thought there might be some gcc-specific tweaks in dSFMT.

                    Code:
                    #include <sys/time.h>
                    #include <iostream>
                    
                    #include <boost/random/linear_congruential.hpp>
                    #include <boost/random/uniform_int.hpp>
                    #include <boost/random/uniform_real.hpp>
                    #include <boost/random/variate_generator.hpp>
                    #include <boost/random/mersenne_twister.hpp>
                    
                    // This is a typedef for a random number generator.
                    // Try boost::mt19937 or boost::ecuyer1988 instead of boost::minstd_rand
                    typedef boost::mt19937 base_generator_type;
                    
                    int main() {
                       long nIt=1L<<32;
                       double x=0;
                    
                       base_generator_type generator(42u);
                    
                       boost::uniform_real<> uni_dist(0,1);
                       boost::variate_generator<base_generator_type&, boost::uniform_real<> > uni(generator, uni_dist);
                    
                       for (long i=0; i < nIt; i++) {
                    	x +=  uni();
                       }
                    
                       std::cout << "x=" << x << '\n';
                    }
                    On my CPU the pathscale compiled code is more than two times slower than GCC 4.3 when executing this code compiled with "-O3 -msse2". By the way, the latest Intel Compiler perfomed similar to GCC.

                    That's why I ran the phoronix suite on my machine, this time GCC 4.5 vs. yesterday's nightly version of the Pathscale compiler.

                    I used the linux-system suite, however not every benchmark compiled. The results are here:

                    http://openbenchmarking.org/result/1...GR-1106168GR02

                    This is what I did to run the benchmarks:

                    Code:
                    export CFLAGS="-O3 -msse2"
                    export CXXFLAGS="-O3 -msse2"
                    export CC=pathcc
                    export CXX=pathCC
                    export FFC=pathf90
                    export F90=pathf90
                    export F95=pathf95
                    export cc=pathcc
                    export cxx=pathCC
                    export ffc=pathf90
                    export f90=pathf90
                    export f95=pathf95
                    
                    rm -Rf $HOME/.phoronix-test-suite/installed-tests/pts/
                    phoronix-test-suite batch-install pts/linux-system
                    phoronix-test-suite batch-run pts/linux-system
                    And then the same with GCC

                    Code:
                    export CFLAGS="-O3 -msse2"
                    export CXXFLAGS="-O3 -msse2"
                    export CC=gcc
                    export CXX=g++
                    unset FFC
                    unset F90
                    unset F95
                    export cc=gcc
                    export cxx=g++
                    unset ffc
                    unset f90
                    unset f95
                    
                    rm -Rf $HOME/.phoronix-test-suite/installed-tests/pts/
                    
                    phoronix-test-suite batch-install 1106168-GR-PATHSCALE46
                    phoronix-test-suite batch-run 1106168-GR-PATHSCALE46

                    Maybe I'm doing something wrong, I don't know. Comments are welcome.

                    Comment


                    • @codestr0m:

                      do you also support other architectures, specifically MIPS (mips32) ?

                      Comment


                      • Originally posted by codestr0m View Post
                        If you think of a compiler as a front-end and a backend then most of the source level compatibility happens at the front-end. In the case of GCC it has also adds a bunch of extensions for developers to use. The path64 front-end is gcc-4.2 which makes the parsing/semantic analysis easy, but the problem comes for handling those extensions on the backend. In some cases it's trivial and we just haven't gotten around to it. (Maybe they are rarely used or worse a stupid idea :P) For the most part we handle the most common ones and as a result it's easier for people to migrate code from GCC to Path64/EKOPath. The linux kernel poses a couple additional challenges
                        Yes, i do tend to think of compilers as both front-end and backend. (hence, why i mentioned "internally, they may be very different". but obviously my language wasn't very concise Thanks for clarifying, not only for myself - but for anyone else following this thread.

                        1) Heavy GNU extension use (We're missing some here)
                        2) Heavy inline asm
                        3) Debugging the bootup process (It's not easy to debug why the SMP built kernel doesn't boot)
                        yup, i am not surprised both 1 and 2 pose issues. there is a lot of code in Linux that does not conform to C standards, and heavily relies on GCC specifics... it would be nice if this wasn't the case, but it is what it is. - and obviously there are some advantages and disadvantages in both cases...

                        slightly off-topic but...

                        does ekopath require having your modules builtin, or can it build initrd okay??? (with ICC they must be built in - unless something has changed). - also by the same token - would having the modules built in, improve the situation / problems related to #3 ???

                        Another example is glibc. In this case making our GCC front-end emit IR that is correct for the backend was a major pita. With ENZO we've made *huge* changes around this area and now a lot easier. (GCC/GNU code is gone entirely and why you may see ENZO building an small linux distribution from the ground up before EKOPath)
                        interesting, I did some reading on ENZO, and I'll be interested in testing that out, when it is available for you guys at ekopath - developing ENZO must be a little more flexible, eh?? (being as it seems to allow a more 'fresh' approach).

                        Thanks for the insights codestrom - i'm sure you can appreciate this, but often it is hard to get the quality of inside information - that you seem to be quite transparent about. it's really great...!

                        cheerz

                        Comment


                        • Originally posted by mlau View Post
                          @codestr0m:

                          do you also support other architectures, specifically MIPS (mips32) ?
                          I know I'm not codestr0m, but although my initial reaction was "lol no", it seems that (depending on which specific bits they open source) MIPS might be a possibility. See http://www.pathscale.com/node/234 for details.

                          Although I would speculate that, based on sheer popularity, they probably put most of their (man-hour) effort into optimizing AMD64 and Intel 64.

                          Comment


                          • Benchmark update

                            Hi!

                            As an uptate to my last posting, I added the results of the intel compiler to the GCC/Pathscale bench.

                            http://openbenchmarking.org/result/1...GR-1106167GR06
                            Code:
                            export CFLAGS="-O3 -msse2"
                            export CXXFLAGS="-O3 -msse2"
                            export CC=icc
                            export CXX=icpc
                            export FFC=ifort
                            export F90=ifort
                            export F95=ifort
                            export cc=icc
                            export cxx=icpc
                            export ffc=ifort
                            export f90=ifort
                            export f95=ifort
                            
                            rm -Rf $HOME/.phoronix-test-suite/installed-tests/pts/
                            phoronix-test-suite batch-install pts/linux-system
                            phoronix-test-suite batch-run     1106167-GR-1106168GR02
                            The intel compiler is mostly in between those two, rarely worse.

                            Thoughts?

                            Comment


                            • Originally posted by leidola View Post
                              Hi!

                              As an uptate to my last posting, I added the results of the intel compiler to the GCC/Pathscale bench.

                              http://openbenchmarking.org/result/1...GR-1106167GR06
                              Code:
                              export CFLAGS="-O3 -msse2"
                              export CXXFLAGS="-O3 -msse2"
                              export CC=icc
                              export CXX=icpc
                              export FFC=ifort
                              export F90=ifort
                              export F95=ifort
                              export cc=icc
                              export cxx=icpc
                              export ffc=ifort
                              export f90=ifort
                              export f95=ifort
                              
                              rm -Rf $HOME/.phoronix-test-suite/installed-tests/pts/
                              phoronix-test-suite batch-install pts/linux-system
                              phoronix-test-suite batch-run     1106167-GR-1106168GR02
                              The intel compiler is mostly in between those two, rarely worse.

                              Thoughts?
                              interesting results. I'm not surprised that ICC doesn't tend to produce worse benchmarks, very often. I remember using ICC with Firefox a long while back (with -ipo) - the difference over gcc was very noticable. however, i now use firefox4 with gcc (-pgo) which is also very good - but i never got to compare the 2 head to head.. a friend of mine uses ICC to compile his libraires for transcoding (i don't really know the specifics) but he swears by it's performance gains.. - ICC is a good compiler - but also comes with some caveats; like it's licensing, which in turn makes it less useful to developers and the open-source community as well. hence, why it doesn't really have much adoption..in this way i think in the long run PathScale's technologies will be much more useful to everyone...

                              beyond that, based on these tests alone - it would seem each compiler tends to have it's own strong points. however, something important to note is -03 isn't always the best flag to use, as it doesn't always produce the fastest binaries.. sometimes it can introduce performance regressions (depending of course, on a number of different factors, probably from compiler to compiler - may vary).

                              a more interesting set of benchmarks would be if each compilation was fine-tuned for each compiler'. however, obviously, this is easier said than done. - I know myself, i have way too little experience with ekopath 4 to be confident enough in my ability, to really exploit it.

                              anyways, that my 2 cents - thanks for posting your results. - great stuff!

                              cheerz
                              Last edited by ninez; 06-16-2011, 07:00 PM.

                              Comment


                              • Originally posted by pszilard View Post
                                When I have a bit of time I'll try and get back with numbers - assuming that i will be able to compile the code.
                                Compiling Gromacs fails in the linking phase with -Ofast.

                                Code:
                                Signal: Segmentation fault in Assembly phase.
                                Error: Signal Segmentation fault in phase Assembly -- processing aborted
                                [backtrace...]
                                pathcc INTERNAL ERROR: /home/pszilard/programs/ekopath-4.0.10-nightly/lib/4.0.10/x8664/be died due to signal 4
                                INTERNAL ERROR: /home/pszilard/programs/ekopath-4.0.10-nightly/bin/pathcc returned non-zero status 1make[3]: *** [src/kernel/mdrun] Error 1
                                The compilation of fftw similarly fails with -Ofast and -ipa seems to be the source of the problem.

                                I guess I should file a bug report.
                                Last edited by pszilard; 06-17-2011, 09:43 AM.

                                Comment

                                Working...
                                X