Announcement

Collapse
No announcement yet.

Linus Torvalds Just Made A Big Optimization To Help Code Compilation Times On Big CPUs

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Linus Torvalds Just Made A Big Optimization To Help Code Compilation Times On Big CPUs

    Phoronix: Linus Torvalds Just Made A Big Optimization To Help Code Compilation Times On Big CPUs

    For those using GNU Make in particular as their build system, the parallel build times are about to be a lot faster beginning with Linux 5.6 for large core count systems. This landing just after the AMD Threadripper 3990X 64-core / 128-thread CPU launch is one example of systems to benefit from this kernel change when compiling a lot of code and making use of many GNU Make jobs...

    http://www.phoronix.com/scan.php?pag...rallel-Job-Opt

  • #2
    Ahhh this Make me so happy!

    Comment


    • #3
      Might be worth throwing in a more typical CPU in the mix during testing, just to see if any difference was made there too.

      Comment


      • #4
        Originally posted by doublez13 View Post
        Ahhh this Make me so happy!
        Ohhhh Make. That was so funny that I’m now happy.

        Comment


        • #5
          As awesome as this is, it makes me kind of sad that we need to see kernel changes in order to improve on Make's poor parallelism.

          Comment


          • #6
            Originally posted by re:fi.64 View Post
            As awesome as this is, it makes me kind of sad that we need to see kernel changes in order to improve on Make's poor parallelism.
            Make has been an exemplar of good parallelism since long before multi-threaded apps were commonplace. This patch is improving kernel performance in a way that Make has no control over. Even Torvalds acknowledges this in the commit message and also that the "locking tokens" pattern used by the Make jobserver is a legitimate (albeit not typical) use of pipes.

            tl;dr what you say is the exact opposite of reality and you clearly have no clue what you're talking about.

            Ninja has some advantages over Make, but they're mostly to do with how it computes which files have changed -- not parallelism.
            Last edited by xinorom; 02-09-2020, 12:33 AM.

            Comment


            • #7
              Now all projects can start reviving their autotools build systems.
              Also, I wonder why people who opposed systemd ("it's not the traditional UNIX way") didn't oppose introducing a new build system based on an obscure new language.
              After all, shell scripts and Makefiles are the traditional UNIX way. :-)

              Comment


              • #8
                Originally posted by zboszor View Post
                I wonder why people who opposed systemd ("it's not the traditional UNIX way") didn't oppose introducing a new build system based on an obscure new language.
                Do you really have to make yet another thread about systemd, despite it being completely off topic...?

                Comment


                • #9
                  Linus and ESR getting their hands dirty within weeks of each other. What's next? RMS writing emacs from scratch a third time?

                  Comment


                  • #10
                    Originally posted by xinorom View Post

                    Make has been an exemplar of good parallelism since long before multi-threaded apps were commonplace. This patch is improving kernel performance in a way that Make has no control over. Even Torvalds acknowledges this in the commit message and also that the "locking tokens" pattern used by the Make jobserver is a legitimate (albeit not typical) use of pipes.

                    tl;dr what you say is the exact opposite of reality and you clearly have no clue what you're talking about.

                    Ninja has some advantages over Make, but they're mostly to do with how it computes which files have changed -- not parallelism.
                    That's not the vibe I got at all:

                    This pattern is fairly traditional on Unix, and works very well, but will waste a lot of time waking up a lot of processes when only a single reader needs to be woken up when a writer releases a new token.
                    It's entirely a legitimate use of pipes, but there's no evidence for it being "good" necessarily.

                    There's an interesting article about the GNU make jobserver here, written by one of the authors: http://make.mad-scientist.net/papers...mplementation/

                    It's interesting to note that the select implementation that got thrown out is exactly what Ninja more efficiently does, but Ninja also has the advantage of not having to run on many old and obscure UNIXes.

                    To be clear, I'm not saying make is being bad here per se, but more that there are better ways of doing this, even e those are ways make itself can't necessarily use.

                    (Another side note: Ninja's performance is due to a lot of difference things, not just file change checks. A lot of it is written out here: https://aosabook.org/en/posa/ninja.html
                    But at minimum, the change detection is pretty standard for a build system that's trying to be efficient. There are other build systems that tend to try far fancier things, such as Tup.)

                    Comment

                    Working...
                    X