Linux 6.3 Improvements Yield Better Chances Of Successfully Compiling The Kernel With ~32GB RAM

[phoronix]

Phoronix: Linux 6.3 Improvements Yield Better Chances Of Successfully Compiling The Kernel With ~32GB RAM

For those doing large Linux kernel builds such as with the "allyesconfig" build option for including as many of the available drivers as possible into the assembled Linux kernel image, objtool improvements ready to go with Linux 6.3 should cut down on the RAM usage and also speed-up the kernel build time. These improvements were motivated by Linux kernel developers beginning to run out of memory when trying to carry out the "allyesconfig" kernel builds on desktops with 32GB of RAM...

https://www.phoronix.com/news/Linux-6.3-objtool

[andyprough]

That's nuts. I've built the kernel many times on an 11-year-old 8gb laptop, including many instances of Linux-libre, the vanilla kernel, and the Liquorix kernel. And browsed the web and ran various vm's while doing it.

[Jakobson]

That's nuts. I've built the kernel many times on an 11-year-old 8gb laptop, including many instances of Linux-libre, the vanilla kernel, and the Liquorix kernel. And browsed the web and ran various vm's while doing it.

With the allyesconfig?

[schmidtbag]

I'm a little confused by this, because isn't RAM usage when compiling correlated to number of threads? I don't see how an 11 year old laptop with perhaps only 8 threads has the potential to use up tens of GB of RAM unless for some reason you set the -j flag to something way larger than 8. Meanwhile, I imagine if you were to compile on something like a 96 core Epyc, you'd be using a lot more than 32GB.

[Berniyh]

I'm a little confused by this, because isn't RAM usage when compiling correlated to number of threads? I don't see how an 11 year old laptop with perhaps only 8 threads has the potential to use up tens of GB of RAM unless for some reason you set the -j flag to something way larger than 8. Meanwhile, I imagine if you were to compile on something like a 96 core Epyc, you'd be using a lot more than 32GB.

12-16 cores aren't uncommon these days even in consumer systems. I doubt that many of those have more than 32 GB, though.
So 20-30 jobs could 32 GB RAM could be a very common thing.
But I don't think that the number of jobs is the problem here.

However, if you do something like allyesconfig (which tbh nobody should ever do other than just testing purposes), things can change.
The memory usage for compiling all the object files is relatively low, even for many jobs, I just checked with -j32, I get less than 3 GB of memory usage during that period.
But keep in mind that in the end you're going to put a lot of stuff into the final image and that's where memory consumption increases (in my test to about 26 GB only due to the compilation). This should also be independent of the number of jobs, because iirc ld is single-threaded, but I didn't check, because I can't be bothered to do an allyesconfig compilation with only 1-4 jobs …

The linking can also be the peak memory usage when you compile a big project like qtwebengine, which is why it can be beneficial to use gold instead of bfd here, due to its lower memory usage. (Not sure about lld, didn't check.)
For the kernel, the gold linker doesn't work, though.

Edit: btw the gzip compressed image is 209 MB big. Not something you want to use.

[sinepgib]

...
Actually very interesting information
...

Do note however schmidtbag is responding to andyprough who mentions he built it in an 11 years old machine, not a current consumer system.
But yeah, the story is, AFAIK, mostly about the linking step in the allyesconfig where you have a humongous number of object files to link.

[kylew77]

I have a question that is maybe a bit off topic but in the OpenBSD world they relink the kernel at each boot across all architectures even single board computers like Raspberry PIs, Alphas, SPARC 64s, etc. Basically computers where 2GB of RAM is A LOT of RAM, and for the most part this basically works there is a flag to disable it on i486 and i586 systems but it mostly works with 2GB of RAM. There are no kernel modules any more in OpenBSD all are built into the kernel. Is there kernel that much smaller than the Linux kernel that it doesn't need 32GB for linking the kernel? Or does actual compilation use more RAM than linking does?

[andyprough]

The memory usage for compiling all the object files is relatively low, even for many jobs, I just checked with -j32, I get less than 3 GB of memory usage during that period.
But keep in mind that in the end you're going to put a lot of stuff into the final image and that's where memory consumption increases (in my test to about 26 GB only due to the compilation). This should also be independent of the number of jobs, because iirc ld is single-threaded, but I didn't check, because I can't be bothered to do an allyesconfig compilation with only 1-4 jobs …

OK, I see now, it's not really a problem to compile a kernel with much less memory, but to create/link the final image of kernel+all-drivers is where the memory would skyrocket. That makes more sense, because I don't recall memory ever being a problem even on lower-spec machines with a typical kernel. Thank you for testing it and reporting your memory usage.

[Brook-trout]

Linux developers would be hailed as heros to come up with a utility that would build a list of modules required based on your hardware and the Software you will be employing. A Linux kernel built and tuned for your machine. Is the best of all in a working environment especially if you are deploying multiple identical hardware computers.

[Berniyh]

I have a question that is maybe a bit off topic but in the OpenBSD world they relink the kernel at each boot across all architectures even single board computers like Raspberry PIs, Alphas, SPARC 64s, etc. Basically computers where 2GB of RAM is A LOT of RAM, and for the most part this basically works there is a flag to disable it on i486 and i586 systems but it mostly works with 2GB of RAM. There are no kernel modules any more in OpenBSD all are built into the kernel. Is there kernel that much smaller than the Linux kernel that it doesn't need 32GB for linking the kernel? Or does actual compilation use more RAM than linking does?

There are a number of things that could play a role. e.g. different linker, different code, amount of drivers (the Linux kernel holds much more drivers than any BSD kernel).

However, keep in mind that also for Linux it's a really special case to hit memory consumption that high.
If you do a typical configuration where most things are modules, you'll hit 1-1.5 GB during linking. Maybe a bit more if you compile a couple of drivers in.
So the title of the news post really is a bit misleading. For most configurations the change won't matter.
Which makes sense, since if it would really matter and many systems wouldn't be able to compile the kernel, the shouting and complaining would be unbearable.