Torvalds Is Unconvinced By LTO'ing A Linux Kernel

[erendorn]

I care. One might ask why you're on a Linux forum if your philosophy is so different from mine.

FTFY.
One can care about linux and still not care about what you specifically care.

[Brane215]

Having such a list publicly documented would help a lot of people: users to avoid using LTO on those, devs to focus on those. You yourself complained about lack of documentation on LTO.

If not a bug, then a gcc wiki page "Problematic packages with LTO" or such.

maybe, but I have a strong feeling they know about those holes.

Besides, this has made me reGoogle sh*t about LTO.

So it sent me back to "GCC Internals Manual", which I either didn't see before or it was too high for me and my brain perceived it as white noise.

Be it as it may, now I can kind of follow it. And it deals with LTO and -fwhopr etc.

I'll try to read through it and see if I can contibute in more useful way after that.

Reading that stuff gives me headaches, but it sure beats long nights guessing my way around without any documentation...

[curaga]

Having such a list publicly documented would help a lot of people: users to avoid using LTO on those, devs to focus on those. You yourself complained about lack of documentation on LTO.

If not a bug, then a gcc wiki page "Problematic packages with LTO" or such.

[Brane215]

I think you posted the list in another thread. But have you posted it to GCC bugzilla?

No. AFAIK they need easily testable small example that exhibits the error. This means that I was expected to sit down, understand what's going on and reduce bazillion files to one of a few lines.

I didn't have the time nor knowledge for such things.

And have I just showed them whole package in the report, what good would that do ? They must have known things aren't 100% peachy since so many packages failed.
And without doing the journey myself I'd just report what they already know - not all packages compile with LTO. And list of those was not shrinking, so they obviously haven't been investing work into this directly and one or a few more package names wouldn't mean a thing.

[curaga]

I've used freshest gcc I could find. Last few gcc bumps in gentoo were mine, since I couldn't wait for an official ebuild. I recompiled everything so many times.

And even with gcc-4.8.2 and freshest binutils, my list of problematic packages was quite long and not shrinking much.

I think you posted the list in another thread. But have you posted it to GCC bugzilla?

[curaga]

I also use Windows 8.1 for gaming and Mac OS X and can sure tell you that the rest of the world doesn't care. Mac apps are huge. Windows games are huge. The binaries are huge. Nothing is optimized well. Windows updater for Java doesn't delete old versions and the C: drive can be filled with 50 versions of Java 5 and 60 versions of Java 6. Nobody cares. On top of that they constantly run antivirus software which slows down the machine.

We care. One might ask why you're on a Linux forum if your philosophy is so different.

[Brane215]

Few more things, worth pointing out:

1. LTO was experimented with before gcc-4.6, it was introduced with 4.6 and by first 4.7 it was declared as pretty complete, practically ready.

From IIRC 4.6.3 to 4.8.2 I compiled each now revison of gcc and binutils in hope that things will get better. I can't remember one problem that I have solved with this. Even those _few_ packages that I managed to recompile with LTO have bitten me in the ar*e later.

2. Have you noticed that even now even here no one actually knows even loosely WTF this is and how it is supposed to work ? It is all more or less anecdotal, at least out of the circles of compiler developers.
Same with flag and their effects. I was told first that LDFLAGS have to contain CFLAGS, so that linker uses same optimisations as compiler did for optimal end effect.
Then someone came along and told me this is totally wrong since one should NOT pass compiler flags to linker and that linker will get the info about the flags used from the object files themselves.

Now hovewer is another day and I am told here that this is wrong and first version was correct all along.
And tommorow is another day, so who knows ?

Sometimes I get the feeling that for this crowd journey (= fiddling with compiling) is more important than the journey (= getting the code of desired quality).

[Brane215]

Well I suppose the issue is more that LTO isn't something meant for using everywhere on a system where things are updated and changed constantly.

Nope. That's exactly the environment that it should do best in - where there are many small things getting compiled into many bigger and those bigger being recomplied now and then. Since LTO can see inside at least fat binaries, it should be able to do its magic.

For a case like that it's probably best to use it for individual things rather than on big dependencies. With something like Android everything is just compiled all at once and isn't updated in pieces etc.

My point was that if this things crop up in such cases, there are serious bugs and inconsistencies in implementation and at least for me, solution is not in using LTO in a way that I simply won't see them.
For my own playing that _might_ be acceptable, but not for something I intend to put in public use or give to customers.

For the kernel though (what this thread was concerning) I don't see why there's any reason not to support LTO. Even if the gains aren't extreme, if we turned away every patch that only offered a 3% speed improvement or 4% size reduction we'd have an extremely slow and stagnated kernel in my opinion. Those 3 and 4 percent gains add up pretty fast.

And WEEKS of lost time, mopping the cr*p afteer each such gain have lost me half of lifetime already. And it is adding up infinitely faster.
Doing it for testing is fine, but using this in kernel for 1% gain on some irellevant test for 99.999% of the planet while risking for the thing to ge berserk and make fruit salad out of one's disk/s/ is IMHO moronic.

[erendorn]

Look at OpenSSL and heatbleed bug. How many people were using it ? For how many years ?

Should we conclude from that that we shouldn't use (or have used) OpenSSL?

[MWisBest]

I've used freshest gcc I could find. Last few gcc bumps in gentoo were mine, since I couldn't wait for an official ebuild. I recompiled everything so many times.

And even with gcc-4.8.2 and freshest binutils, my list of problematic packages was quite long and not shrinking much.

Worse yet, what compiled with LTO was not always repeatable. Some packages that compiled initially, all of the sudden would fail to recompile after some time. It had something to do with the sequence in which it was complied, copared to its dependencies, and much of that effect was RECURSIVE.

In theory, LTO is just great and practically functionally equal to the classic way.

In reality, compile would often fail at the final link where either:

- linker would spew out crap that even Google has never seen

- it would say scary things like that it has XY different definitions for function W

- that it can't find function W to link to

- that function W it is seeking for and the one found are not compatible


And WRT to your proof by numbers of WHOLE 1000 users for 5 months, it's pathetic.

Look at OpenSSL and heatbleed bug. How many people were using it ? For how many years ?

Well I suppose the issue is more that LTO isn't something meant for using everywhere on a system where things are updated and changed constantly. For a case like that it's probably best to use it for individual things rather than on big dependencies. With something like Android everything is just compiled all at once and isn't updated in pieces etc.
For the kernel though (what this thread was concerning) I don't see why there's any reason not to support LTO. Even if the gains aren't extreme, if we turned away every patch that only offered a 3% speed improvement or 4% size reduction we'd have an extremely slow and stagnated kernel in my opinion. Those 3 and 4 percent gains add up pretty fast.

WRT Proof: http://forum.xda-developers.com/gala...-2014-t2427087
I don't like bragging or boasting, the only reason I brought up the numbers there was because it was relevant to my case. There are even builds that have accumulated nearly 2000 downloads, however as of late the download numbers have dropped as the amount of time I've had for FML has dropped too, but I don't care, I actually have a good attitude towards it: "If they aren't happy with FML, I would prefer they give other ROMs a shot until they find something they enjoy, and/or give me some feedback as to what I could improve upon in FML."

The OpenSSL bug was with code, it did pop into my mind though.