At Least 27% Of Gentoo's Portage Can Be Easily LTO Optimized For Better Performance

[In_Between_Names]

It's however true. While it is part of GCC for a long time now has it fallen into bit rod, because the original author has stopped maintaining it. For a while were other GCC developers only collecting bug reports on it, but not fixing any of them. See the Graphite Meta Bug report. A few of these bugs got fixed recently, but there are still many open bugs. Without anyone actively maintaining the code and it only being dragged on might the devs eventually decide to drop it. It's sad, but there is nothing one can do other than to become active oneself.

Indeed, Graphite needs work. Were more people to use it, more work would probably be done on it. In any case, users of GentooLTO are under no obligation to use Graphite. I provide easy instructions in the overlay such that you can customize your flags to your heart's content.

[emerge-e-world]

Can a Gentoo user answer a quick question for me -- Can Gentoo be installed/compiled up to a working desktop from a chroot on another Linux desktop (Manjaro in my case)?

Absolutely. In fact, it works the same way as if you were setting up a gentoo installation from a LiveCD. It does not have to be a gentoo system, all you need is some sort of working linux system that is capable of entering the chroot, give you network access and things like that.

What you do is this: Create a partition to become your gentoo root fs, put a filesystem on it, mount it somewhere. (if you're using btrfs, you can even share your existing root fs, by putting gentoo into a subvolume.) Then extract a stage3 tarball onto it (get it from https://www.gentoo.org/downloads/), setup a few things and chroot into it.

The few things thing to setup before chroot'ing are things like: copy over your /etc/resolv.conf to make DNS work, bind mount /dev /sys and /proc so that portage and certain packages can query things from your kernel etc. Nothing that complicated, here is a quick overview over the steps in more detail: https://wiki.gentoo.org/wiki/Chroot

If you've never built a gentoo box before, you should look at the install guide to learn more about how the install process works, and espeically to find out how to get from your chroot to something bootable (https://wiki.gentoo.org/wiki/Handbook:AMD64). You can skip the first 3 steps, since you already have a running OS, start at "preparing the disks".

[skeevy420]

Absolutely. In fact, it works the same way as if you were setting up a gentoo installation from a LiveCD. It does not have to be a gentoo system, all you need is some sort of working linux system that is capable of entering the chroot, give you network access and things like that.

What you do is this: Create a partition to become your gentoo root fs, put a filesystem on it, mount it somewhere. (if you're using btrfs, you can even share your existing root fs, by putting gentoo into a subvolume.) Then extract a stage3 tarball onto it (get it from https://www.gentoo.org/downloads/), setup a few things and chroot into it.

The few things thing to setup before chroot'ing are things like: copy over your /etc/resolv.conf to make DNS work, bind mount /dev /sys and /proc so that portage and certain packages can query things from your kernel etc. Nothing that complicated, here is a quick overview over the steps in more detail: https://wiki.gentoo.org/wiki/Chroot

If you've never built a gentoo box before, you should look at the install guide to learn more about how the install process works, and espeically to find out how to get from your chroot to something bootable (https://wiki.gentoo.org/wiki/Handbook:AMD64). You can skip the first 3 steps, since you already have a running OS, start at "preparing the disks".

Thanks. That's kind of what I was thinking, but, since the handbook goes as far as getting the kernel installed, rebooting, and going on from there, I never wanted to risk attempting a straight up nothing to desktop Gentoo install without at least asking first.

I've attempted building Gentoo on some of my previous PCs, but usually around 3 days in something would break and doing it all from a terminal and a phone was a little discouraging...should have just followed my gut and did it all from a chroot but I was following the handbook to the letter...and not given up so easily...

The experience game me a lot of respect for the ones who did it all on older single and dual core systems for weeks at a time. Now with at least 150% more IPC, 4x the number of threads, & 6x the ram I expect it'll go a little faster this time .

[tildearrow]

Very late typo:

While they survey is user-driven and not systematically testing all available packages,

[emerge-e-world]

I've attempted building Gentoo on some of my previous PCs, but usually around 3 days in something would break and doing it all from a terminal and a phone was a little discouraging...should have just followed my gut and did it all from a chroot but I was following the handbook to the letter...and not given up so easily...

yeah not having a GUI with a browser at hand sounds very tedious. That's how I did my first gentoo builds as well, although there were no smart phones back then. But I think I had a laptop at hand while I setup my desktop.

Doing it in chroot's is how I'm always doing it these days. Especially when I want to bootstrap for a slower mashine, I tend to just build my root fs in a chroot on my desktop or server with lots of cores, then make a tarball of it and extract it onto the target mashine. If you're afraid of messing something up, it's quite useful to build it on btrfs and take snapshots in between the steps. That way you can always go back.

The experience game me a lot of respect for the ones who did it all on older single and dual core systems for weeks at a time. Now with at least 150% more IPC, 4x the number of threads, & 6x the ram I expect it'll go a little faster this time .

ahh, the good old times. Yes, rebuilding the world could take days/weeks back in the day. Packages were smaller, but building things like libreoffice on a single core cpu, that was fun for more than a day. These days you can rebuild a normal desktop installation in about a day with an average desktop cpu. But hey, ever needed an excuse to buy a Threadripper? I couldn't think of a better use case

[ms178]

Definitly not. But as a data point, Martin Liška has just finished experimental rebuild of OpenSUSE with GCC 9 and GCC 9 and LTO. Switching GCC8 to GCC9 increases binary sizes 8.42GB to 8.52GB (1.14%). I am looking into those packages that grew - it seems to me mostly for lapack, VLC and similar performance optimized stuff due to the fact that inline limits was increased in GCC9 to help performance of modern C++ stuff.

Enabling LTO reduces size to 3.92GB->3.71GB. So about 5% reduction. Note that the size includes things not built with GCC such as go which accounts for some of largest binaries on the distro.

Some large packages did not use system flags and we will need to rebuild those, so the differences will hopefully improve still. Only go, postgress and libqt-declarative which clearly disabled LTO accounts to 400MB of binaries and overall 700MB of binaries did not seem to change at all and thus needs to be looked at.

LTO also reduces debug info from 27.56GB to 22.73GB (17%). We will also look into the performance

Thanks for providing this update. Does the rebuild provide a chance to try more aggressive compiler flags like Graphite, PGO or is the current focus on LTO only?

[Grinch]

Does the rebuild provide a chance to try more aggressive compiler flags like Graphite, PGO or is the current focus on LTO only?

PGO requires you to run the program to generate runtime data which is then used to guide the optimizations, this is not feasible to do manually on all the packages across a whole distro. A few select applications has automated functionality for PGO 'out-of-the-box', like Firefox, x264, but the vast majority of packages do not.

As for Graphite, I have never had any worthwhile performance gains from it myself on the stuff I regularly benchmark (archivers, encoders, emulators, renderers).

[PeeJay]

I was one of the survey respondents - rarely had any trouble with it (that wasn't self inflicted!). One bug I submitted a patch to turn off lto, another one the maintainer did within a few days.
I would be interested to know if those 5000 packages cover the 90% most used packages on Gentoo. I'm guessing so. 27% coverage doesn't seem like much, but on my desktop machine I have all sorts of weird stuff installed with a full kde desktop and it's only around 700 packages.

[RealNC]

Indeed, Graphite needs work. Were more people to use it, more work would probably be done on it. In any case, users of GentooLTO are under no obligation to use Graphite. I provide easy instructions in the overlay such that you can customize your flags to your heart's content.

It should not be enabled by default.

[sdack]

It should not be enabled by default.

No, on the contrary! When you want to help users then you don't even point them to Gentoo, but to Ubuntu and the likes. This is a Gentoo thing and anyone with a clue will know which options are save to use and which are not. Those who don't are out of their depth anyway just by using Gentoo. You cannot help everyone, but if you try anyway and there only start at the Graphite options then you're doing it wrong, too.

So does -ffast-math (or -Ofast) as well as -fipa-pta need to be enabled by default, too. Anything that doesn't run with these options enabled then needs to be patched so it disables the options specifically for just the one package.