Linus Torvalds Just Made A Big Optimization To Help Code Compilation Times On Big CPUs

[coder]

sdack - interesting. The -l option existed before the jobserver, and it never worked well because load average updates too slowly. By the time it hits your threshold your machine is already thrashing.

When I implemented a parallel execution framework (i.e. for running test cases), I dealt with this by tracking jobs I spawned and augmenting the load average by the complement of each job's estimated impact on the load. This gave me the effect of instantly knowing the current system load.

It works perfectly, so long as the baseline load of the machine is otherwise stable. However, if you had multiple runs (either by the same or different users), then it would hose the machine, since each did not know about the others.

The notion of multiple users running a build process on a single machine is pretty alien these days. Everything is VMs and containers now, with explicitly allocated CPU resources.

We still share native hardware between users, at my job, due to the ease and efficiency benefits of doing so.

[coder]

My comment was not about systemd, it was about double standards.

Your comment was about trolling and/or demonstrating a lack of understanding of the full range of grievances with systemd.

Off-topic, in either case.

[skeevy420]

On my own C compiles I usually use -j NCPU*1.5 which keeps the CPUs 100% busy even if a compile is busy with I/O. Spawning much more than that tends to just waste CPU thrashing between processes. On C++ compiles I've found that I can't go above NCPU, either because I don't have enough RAM to handle that many C++ instances, or some other bottleneck overrides.

I have enough ram I've never ran into that problem on the few things that I compile locally these days. I've had 16+ GB of ram for so long that a lot of times I don't even consider low memory situations.

[coder]

i recall attempt at kernel build system rework in really python-based tool by esr. it didn't fly because of slowness, not because of dependency

SCons? I tried it, once; wasn't impressed.

[coder]

On my own C compiles I usually use -j NCPU*1.5 which keeps the CPUs 100% busy even if a compile is busy with I/O.

Even on hard disk-based builds, I've simply not needed this. Of course, I'm compiling C++, which is more compute-intensive.

On C++ compiles I've found that I can't go above NCPU, either because I don't have enough RAM to handle that many C++ instances, or some other bottleneck overrides.

More RAM will also help with your I/O bottlenecks, due to more read caching & write buffering. It's probably the main reason I can just use -j N*Cpu. RAM is cheap (if we're talking about DDR4), but prices are expected to rise, this year.

I have enough ram I've never ran into that problem on the few things that I compile locally these days. I've had 16+ GB of ram for so long that a lot of times I don't even consider low memory situations.

I've needed up to 1 GB per thread, when compiling some C++. Better-written code usually doesn't require that much, but optimization options are also a factor. Anyway, I wouldn't consider 16 GB to be sufficient, if you were using a 12-core/24-thread Ryzen 9 3900X.

I still recall my horror, when I first noticed that my build on an 8-thread machine with 6 GB was thrashing. This was a first-gen i7, when 6 GB was still a respectable amount of RAM and I was still using a spinning hard disk.

[doublez13]

Have anyone tried to get Linux to build with Meson instead of GNU Make?
Should Linux stay with GNU Make or should it switch over to Meson?

Fuck I don't think that'll ever happen. The kernel build system is so complex.

How about using ninja 😋

[pal666]

In a properly written Makefile, you just make those foreign tools serial dependencies so Make doesn't try to parallelize around them.

sure you can do that but what if those tools have more jobs to do than make itself? also they are not necessarily children of make. and make invocations aren't necessarily children of them

[pal666]

It's only unfortunate to those who run idle processes while requiring maximum CPU performance

isn't that the purpose of running idle processes?

and then don't know how to shut them down.

if you know how to do everything by hand, why do you need make in the first place?

[sdack]

isn't that the purpose of running idle processes?

No. They are merely a compromise. I would first try to get rid of idle processes instead of having them run idle. Save some energy and reduce the CO2 output instead of tasking your CPUs to constantly work when they might not really have to.

Compile jobs then don't only use CPU, but they have to wait for I/O to complete, too. This leaves a small chance for an idle schedule process to get some CPU time and causes small waits on its own. When you want maximum performance then you'll have to get rid of idle processes.

Linux offers a variety of scheduling algorithms, and while having them means having options and choices available, does it often end being a suboptimal solution when you start mixing them together.

If you still think you need to have idle processes running while running parallel builds with Make then just try to give it a higher number for the load count and hope that it kicks all the idle processes out.

[ed31337]

I have enough ram I've never ran into that problem on the few things that I compile locally these days. I've had 16+ GB of ram for so long that a lot of times I don't even consider low memory situations.

Well, that's nice for you, but when I'm compiling the Chromium sources, as a component of Qt, on my 4GB Raspberry Pi 4, I've seriously had problems with running out of RAM.

For whatever reason, the Chromium source code seems to use tons of RAM, building up huge source files called jumbo_XXXX.cpp that totally thrash if I don't limit the number of CPU cores used. And Chromium insists on using ninja instead of GNU make. With ninja, there is no way to limit the number of cores as far as I can tell -- ninja just automatically uses as many cores as you've got, regardless of whether you've got enough RAM.

The only way I manage to build successfully is by manually watching what's going on with top and then sending a SIGSTOP/SIGCONT to excess compilation processes. This allows me to force ninja to only have one or two processes actually running at any one time when it gets into those jumbo source files.

BIG props to Linus for fixing the thundering herd problem with make. By avoiding waking all processes, maybe it'll thrash less hard on low memory systems, besides benefiting those with their huge 64 core processors and infinite RAM.