Torvalds Is Unconvinced By LTO'ing A Linux Kernel

[hubicka]

Actually -march=native is even more problematic than the LTO (depends on platform tho), for example, it generates sse instruction which make context switches slower since it takes a while to set them up, which is pretty bad, especially if you have realtime requirements.

LTO seems much safer, though any gain for desktop users is doubtful due to the fact that distribution kernels are super robust and have everything in modules, which is presumably least effective use-case for LTO (and yet, most widely used).

For fully standrad compliant software, LTO is fully transparent, and you can just enable -flto and expect improvements. Many of key packages (glibc, kernel, web browsers,...) however do a lot of non-standard things and needs some care to work with LTO. So yes, blind rebuild of your Gentoo with -flto is going to show interesting problems, but it really depends on upstream developers of these packages how quickly they will disappear.

LTO is new and has issues, this is a chicken-egg problem we need to solve - because LTO now works well in tests and environment, the problems won't be hammered out effectively without feedback. The more LTO users are here, the faster it will mature.

[curaga]

I understand that 5% off is important in 4 MB flash storage. However in desktop apps it doesn't matter at all. Hard drives are now 1 TB (ssd) and 4 TB (3.5" hdd). You can also set up raid6 or zfs. So you get tens of terabytes and it's very cheap. You shouldn't bother with binary sizes. In fact there's plenty of room for more functionality in Firefox. Luckily they're working hard at implementing more new features with each release.

Of course it matters. In Firefox's case, it directly affects startup speed.

[caligula]

Of course it matters. In Firefox's case, it directly affects startup speed.

But how much it really matters? Firefox static initialization takes a while, especially if you have bookmarks and history and some old session. SSD can read disk 600 MB/s and the binary is smaller than 300 MB. To me the real bottleneck is elsewhere.

[Garp]

So someone has to make tests and benchamarks to evaluate its capabilities, problems and advantages.

I think this is being missed in all the fuss. This is such a basic and simple step in getting software from 'hobby code' to 'production worthy'. Given the scope of the changes, everything needs to be fully tested, and changes on this scope and potential complexity should be fully justified with compelling arguments for rather than vague and rather hand-wavey "it does stuff quicker/smaller".

[Garp]

Looking through that original email thread, even the proponents are saying it's going to result in negligble speed benefits, mostly just some size reduction, largely dependent on how limited your kernel config is (LTO seems to be good at not compiling unnecessary code from disabled code paths)

A few choice quotes:

LTO shows promise for allowing more automation in configuration
handling (that is, requiring less CONFIG options).

People should definitely be warned off using this in any production
setting, but I think it's valuable for developers experimenting with
tiny-size systems to have this easily available in mainline.

4) LTO brings noticeable performance wins on average, but it is largely benchmark
dependent; some see huge improvemnts, others no improvements at all.

Basic observation is there is not much that LTO can do that can not be done
by hand. Careful developer can just identify the important spot and
restructure the soruces.
The runtime benefits are more visible on bigger, bloated and less
optimized projects than on hand tuned video encoder implementation.
I believe Kernel largely falls into hand tuned category despite its size.
...
6) LTO will pay back more in long term.

It is not only because LTO implementation in GCC (and LLVM) has more room
for improvement than the per-file optimizers.

Main things is that despite the aim to be transparent to user,
LTO is invasive change. Existing programs was developed and tuned for
per-file optimization model and many of them contains a lot of hacks to
work around its limitation (such as a lot of inline code in headers, etc.)
With LTO becoming mainstream, developers will have time to work on different
hacks

> I would be curious about the results on Kernel.

We saw some upsides in performance with some standard tests, but nothing
too significant.

At the momen this seems like something certainly interesting, but a bunch of changes without a real payoff at the moment as the tooling and maturity of LTO compilers/optimisers isn't there yet.

[Guest]

For fully standrad compliant software, LTO is fully transparent, and you can just enable -flto and expect improvements. Many of key packages (glibc, kernel, web browsers,...) however do a lot of non-standard things and needs some care to work with LTO. So yes, blind rebuild of your Gentoo with -flto is going to show interesting problems, but it really depends on upstream developers of these packages how quickly they will disappear.

LTO is new and has issues, this is a chicken-egg problem we need to solve - because LTO now works well in tests and environment, the problems won't be hammered out effectively without feedback. The more LTO users are here, the faster it will mature.

I don't see how this is in any way related to my comment, which talked about LTO in kernel and not specific use cases/distributions/userspace stuff.

[Guest]

But how much it really matters? Firefox static initialization takes a while, especially if you have bookmarks and history and some old session. SSD can read disk 600 MB/s and the binary is smaller than 300 MB. To me the real bottleneck is elsewhere.

But that really isn't the point of this discussion, is it.
LTO in kernels case probably only really makes sense for release+static builds (since more stuff is compiled in via modules, the slower will LTO compilation be and any significant gains will be reduced).
Either way, LTO on desktop doesn't probably make much difference, while it could be crucial on embedded (routers, smartphones...).

[OT]
This is true, loading firefox profile from harddrive is most time consuming operation (depending on disk speed taking up to 20 seconds vs a second or two needed for warm start with hot page cache).

This is however true for pretty much any software so it shouldn't really bother us.
One could make argument that we need to invest resources into filesystem performance, for example btrfs with fast realtime compression algorithms such as lz4 or googles snappy.

Another fix is similar userspace solution, upx (upx.sourceforge.net)
[/OT]

[MWisBest]

To the people saying LTO is "unstable" and "breaks things" etc., I'm compiling Android with all but 2 sections of it having LTO enabled, not a single problem from it. LTO isn't simply "experimental" anymore.

[Brane215]

To the people saying LTO is "unstable" and "breaks things" etc., I'm compiling Android with all but 2 sections of it having LTO enabled, not a single problem from it. LTO isn't simply "experimental" anymore.

To all the people that have to avoid sugar and on periodic insuline shots (aka "diabetics") I say that I just downed large Coke over enormous piece of pie. And now I feel just fine.

Also, I could never understand all this talk about contraception.

I never wore or used protection and still managed to walk away without getting pregnant. Which would in my case automatically mean caesarian section, since my penis would pose impossible bottleneck.


Has it ever crosed your mind that we don't run the same harware in comparable circumstances ?

And BTW, just because you haven't seen any problems _yet_ , it doesn't mean that they are not lurking hidden.
That "hey, it works!" initial stage was my experience, too. And then problems kept cropping, so after much headscratching I went into "There are problems, but I'm good with workarounds". And then I figured out that I've just put many hours into it with new problems keep popping up and many things not working in consistant ways.
So, after more than a year, I ended it in "F**k it, it simply aint worth it."

Give it time.

[Brane215]

For fully standrad compliant software, LTO is fully transparent, and you can just enable -flto and expect improvements. Many of key packages (glibc, kernel, web browsers,...) however do a lot of non-standard things and needs some care to work with LTO. So yes, blind rebuild of your Gentoo with -flto is going to show interesting problems, but it really depends on upstream developers of these packages how quickly they will disappear.

LTO is new and has issues, this is a chicken-egg problem we need to solve - because LTO now works well in tests and environment, the problems won't be hammered out effectively without feedback. The more LTO users are here, the faster it will mature.

1. kernel, glibc and other non-standard packages are not that insignificant. Kernel doesn't link to anything outside in userland ( except perhaps that one link for getting accurate time, but that's insignificant), but it needs stability more than anything else in userland. And with LTO option being in testing, I don't see how can the result be trusted atm for anything except testing purposes.

Glibc and similar libraries are not insignificant. If they miscompile with LTO, they'll influence just about anything. And if you compile them non-lto then you are missing big part of the point - big parts will end up being opaque islands to any lto optimisation efforts.

2. I had to do it blindly, because there simply was no one to turn to. CFLAGS & LDFLAGS I used were given to me IIRC at gentoo forum. Someone said that it is only important to turn on optimisation and that linker only does -O1 anyway. They also said that there is no need to repeat CFLAGS while linking and that this was needed only on early lto version of gcc.

IOW, there is not much publicly known or accessible documentation on-line about gcc and related tools. Every now and then someone utters something and this gets picked up by ignorant crowd and praised without detailed knowledge about it.
Had Christianity had so fu**ed up, obscure, hard to get, fragmented and contradictory documentation on "Word of God", whole thing would ended with Stallman, I mean Christ as initiator.