Announcement

Collapse
No announcement yet.

Linus Torvalds Just Made A Big Optimization To Help Code Compilation Times On Big CPUs

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • #51
    Originally posted by ed31337 View Post

    Well, that's nice for you, but when I'm compiling the Chromium sources, as a component of Qt, on my 4GB Raspberry Pi 4, I've seriously had problems with running out of RAM.

    For whatever reason, the Chromium source code seems to use tons of RAM, building up huge source files called jumbo_XXXX.cpp that totally thrash if I don't limit the number of CPU cores used. And Chromium insists on using ninja instead of GNU make. With ninja, there is no way to limit the number of cores as far as I can tell -- ninja just automatically uses as many cores as you've got, regardless of whether you've got enough RAM.

    The only way I manage to build successfully is by manually watching what's going on with top and then sending a SIGSTOP/SIGCONT to excess compilation processes. This allows me to force ninja to only have one or two processes actually running at any one time when it gets into those jumbo source files.

    BIG props to Linus for fixing the thundering herd problem with make. By avoiding waking all processes, maybe it'll thrash less hard on low memory systems, besides benefiting those with their huge 64 core processors and infinite RAM.
    You can disable jumbo builds with the configure argument -no-webengine-jumbo-build, and you can restrict the process NINJA uses by setting the environment variable NINJAFLAGS before configuring qtwebengine. It like make can take -j and -l arguments to limit it.

    But I would recomment cross-building to a Pie from a proper desktop, crossbuilding Qt isn't that hard.

    Comment


    • #52
      Originally posted by carewolf View Post
      You can disable jumbo builds with the configure argument -no-webengine-jumbo-build, and you can restrict the process NINJA uses by setting the environment variable NINJAFLAGS before configuring qtwebengine. It like make can take -j and -l arguments to limit it.
      Thank you very much for that. I might have to give that a go next time I end up compiling Qt again someday.

      But I would recomment cross-building to a Pie from a proper desktop, crossbuilding Qt isn't that hard.
      Maybe if I was in the business of compiling Qt often as a Qt developer. But in my case, I'm just a freelance user of Qt that just wants to have a recent build of Qt on my RPi4. For me, it's a lot easier to just build stuff on the Pi and let it run overnight than fool around with setting up a cross-build environment.

      Plus, my x86 Ivybridge laptop isn't all that great anyway -- sometimes overheats and spontaneously reboots. RPi4 may be slow, but it gets the job done and doesn't overheat or spontaneously reboot, ever.

      Comment


      • #53
        Originally posted by sdack View Post
        No. They are merely a compromise. I would first try to get rid of idle processes instead of having them run idle. Save some energy and reduce the CO2 output instead of tasking your CPUs to constantly work when they might not really have to.
        we are probably talking about different kinds of idle processes. i'm talking about SCHED_IDLE scheduling policy. i use it for boinc. those processes aren't idle, they are just using cpu time which no other process needs.
        Originally posted by sdack View Post
        If you still think you need to have idle processes running while running parallel builds with Make then just try to give it a higher number for the load count and hope that it kicks all the idle processes out.
        and i can use -j16 instead of -j`ncpu` but then i will have to change it between machines or upgrades, which is like anti-automation, i.e. anti-make

        Comment


        • #54
          Just though I should mention that Wendell from Level1Techs did a video on this: https://www.youtube.com/watch?v=zMhv82vQcb0

          Comment

          Working...
          X