At Least 27% Of Gentoo's Portage Can Be Easily LTO Optimized For Better Performance

[Zan Lynx]

LTO will not inline code from shared libraries. Static libraries, yes. But a static link has to be rebuilt to get updates for the library anyway so LTO makes no difference there.

[In_Between_Names]

I am a Gentoo user and I don't see how LTO can be made stable on a rolling release Linux distribution without detailed records about what gets inlined where by the compiler during LTO. If A is using library B and B is using C, it means that the internal code from B and C might end up in A. If the user updates library C, the package manager must be able to detect that it needs to rebuild A and B depending on where the compiler decided to inline C into. The most pessimistic approach (without any tracking of what the compiler actually inlines where) is for rebuild_C to always trigger rebuild_B and rebuild_A. There can be longer dependency chains like A -> B -> C -> D -> E. This also implies that upgrade_GCC should automatically trigger rebuild_allpackages.

I don't see any indication that GentooLTO is a serious mathematically correct attempt to automate handling of LTO dependencies by tracking the inlining of code&data in a rolling release Linux distribution. If somebody was able to find such an indication, please post it here.

Maintainer of GentooLTO here. I think this is a misconception about what LTO actually does. Firstly: in the cases where a shared object is linked against, inlining may not occur because the compiler's IR is not made available in the shared object. Therefore, the ABI used in the non-LTO linking case is used instead, and standard update rules apply.

Secondly: if static linking is used, then of course inlining can occur across the library boundary. However, since it is inlined (by definition), any change to the underlying static library would not have any effect in the linked binary automatically, and to pick up those changes a re-link is necessary. However, this is a well known phenomenon that applies even in non-LTO linking -- for example, when a security advisory hits a linked against static library, you're going to want a recompile regardless of LTO.

In practice, static libraries aren't used that often, and when they are, Portage tracks that dependency for you and cautions you to rebuild, although you are under no obligation or pressure to actually do that. It's not like your system is going to stop working if you're linked against an older static library than the current one. Any used code was included directly in the binary anyway.

[fuzz]

30 users isn't much, though.

30 survey respondents... I didn't even know there was a survey for a report and I've been using this for quite a while.

[fuzz]

Maintainer of GentooLTO here.

FYI, I have an old Asus EEE PC I'm going to install Gentoo on. Going to use -Os and might as well try GentooLTO. Should be interesting .

[hubicka]

LTO is not only about size though.

Definitly not. But as a data point, Martin Liška has just finished experimental rebuild of OpenSUSE with GCC 9 and GCC 9 and LTO. Switching GCC8 to GCC9 increases binary sizes 8.42GB to 8.52GB (1.14%). I am looking into those packages that grew - it seems to me mostly for lapack, VLC and similar performance optimized stuff due to the fact that inline limits was increased in GCC9 to help performance of modern C++ stuff.

Enabling LTO reduces size to 3.92GB->3.71GB. So about 5% reduction. Note that the size includes things not built with GCC such as go which accounts for some of largest binaries on the distro.

Some large packages did not use system flags and we will need to rebuild those, so the differences will hopefully improve still. Only go, postgress and libqt-declarative which clearly disabled LTO accounts to 400MB of binaries and overall 700MB of binaries did not seem to change at all and thus needs to be looked at.

LTO also reduces debug info from 27.56GB to 22.73GB (17%). We will also look into the performance

[In_Between_Names]

They're using Graphite, which is unmaintained, abandoned, and nobody cares about it. This doesn't exactly inspire confidence for the people behind this overlay...

If that's the case, then why is it still receiving patches upstream? It's still receiving love in the GCC bug tracker:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86865

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=89182

This meta-bug tracks all Graphite issues, and was most recently updated this year: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=graphite

There are real documented cases with GentooLTO where we found ICEs when using Graphite and in later versions those ICEs were fixed.
It may not be as actively developed as other parts of the optimizer, but I firmly believe that if it's in the GCC code base, then it should be tested. At the very least, if Graphite is as
lacking in value as you imply, it will provide motivation for removing it from GCC, or otherwise documenting it as something that isn't to be used.
This is entirely consistent with the goals of the overlay.

[sdack]

... unless you explicitly tell the compiler to statically link external libraries, they won't be part of the final binary just because you use LTO.

This is correct.

Most software packages that build with -flto will usually end up only link-time optimising their own source code. Due to most libraries being shared libraries does external code almost never get inlined, and for the reasons mentioned.

Because of this is LTO not used to its fullest, theoretical potential and therefore is often not as effective as people wish for. Most developers know their own code well enough that it runs reasonably well without LTO and their code doesn't require any further inlining. Thus do such projects benefit little from LTO and all it does there is to prolong the compile time. In some cases does LTO then backfire and produces slower code. Yet can LTO still squeeze out a benefit in some cases, when a project has become so large that developers can no longer comprehend its entire code base and as a consequence have to program more carefully and pessimistically, allowing LTO to deliver some significant benefits.

While the future of optimising compilers is found with LTO do I have my doubts if it currently makes sense to compile 30%+ of any distribution with it. Even with GCC 9, where LTO received a few improvements such as better heuristics, is the benefit of using it still marginal and will continue to backfire on many projects.

Anyhow, at the very least is the effort a good stress-test for the compilers.

[FishB8]

They're using Graphite, which is unmaintained, abandoned, and nobody cares about it. This doesn't exactly inspire confidence for the people behind this overlay...

None of that is true. It's been mainlined since forever and is actively maintained. I use it all the time and it works just fine.

[sdack]

None of that is true. It's been mainlined since forever and is actively maintained. I use it all the time and it works just fine.

It's however true. While it is part of GCC for a long time now has it fallen into bit rod, because the original author has stopped maintaining it. For a while were other GCC developers only collecting bug reports on it, but not fixing any of them. See the Graphite Meta Bug report. A few of these bugs got fixed recently, but there are still many open bugs. Without anyone actively maintaining the code and it only being dragged on might the devs eventually decide to drop it. It's sad, but there is nothing one can do other than to become active oneself.

[skeevy420]

Can a Gentoo user answer a quick question for me -- Can Gentoo be installed/compiled up to a working desktop from a chroot on another Linux desktop (Manjaro in my case)?

I've always wanted to use Gentoo due to how much stuff I build on Arch/Manjaro from source but I don't want to have any downtime during the initial install and I'd like to have access to a web browser for the handbook w/o using lynx and multiple virtual terminals because, well, Firefox and Yakuake are handy.