Announcement

Collapse
No announcement yet.

Torvalds Is Unconvinced By LTO'ing A Linux Kernel

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • #46
    Originally posted by MWisBest View Post
    That's a ridiculous comparison...

    You weren't using a recent enough toolchain. I said it isn't experimental ANYMORE. If you're using binutils 2.19 and GCC 4.7, you're not going to have as great of an experience with LTO as you would with binutils 2.23 and GCC 4.8.

    I've been using LTO like this for... jeez, 5 months or so now, with at least a new build every week, used by roughly 1000 other people. No problems have arose from LTO, and if they do after 5 months, it can't be extremely significant and it's not like it's impossible to fix bugs.
    I've used freshest gcc I could find. Last few gcc bumps in gentoo were mine, since I couldn't wait for an official ebuild. I recompiled everything so many times.

    And even with gcc-4.8.2 and freshest binutils, my list of problematic packages was quite long and not shrinking much.

    Worse yet, what compiled with LTO was not always repeatable. Some packages that compiled initially, all of the sudden would fail to recompile after some time. It had something to do with the sequence in which it was complied, copared to its dependencies, and much of that effect was RECURSIVE.

    In theory, LTO is just great and practically functionally equal to the classic way.

    In reality, compile would often fail at the final link where either:

    - linker would spew out crap that even Google has never seen

    - it would say scary things like that it has XY different definitions for function W

    - that it can't find function W to link to

    - that function W it is seeking for and the one found are not compatible


    And WRT to your proof by numbers of WHOLE 1000 users for 5 months, it's pathetic.

    Look at OpenSSL and heatbleed bug. How many people were using it ? For how many years ?

    Comment


    • #47
      Originally posted by Brane215 View Post
      I've used freshest gcc I could find. Last few gcc bumps in gentoo were mine, since I couldn't wait for an official ebuild. I recompiled everything so many times.

      And even with gcc-4.8.2 and freshest binutils, my list of problematic packages was quite long and not shrinking much.

      Worse yet, what compiled with LTO was not always repeatable. Some packages that compiled initially, all of the sudden would fail to recompile after some time. It had something to do with the sequence in which it was complied, copared to its dependencies, and much of that effect was RECURSIVE.

      In theory, LTO is just great and practically functionally equal to the classic way.

      In reality, compile would often fail at the final link where either:

      - linker would spew out crap that even Google has never seen

      - it would say scary things like that it has XY different definitions for function W

      - that it can't find function W to link to

      - that function W it is seeking for and the one found are not compatible


      And WRT to your proof by numbers of WHOLE 1000 users for 5 months, it's pathetic.

      Look at OpenSSL and heatbleed bug. How many people were using it ? For how many years ?
      Well I suppose the issue is more that LTO isn't something meant for using everywhere on a system where things are updated and changed constantly. For a case like that it's probably best to use it for individual things rather than on big dependencies. With something like Android everything is just compiled all at once and isn't updated in pieces etc.
      For the kernel though (what this thread was concerning) I don't see why there's any reason not to support LTO. Even if the gains aren't extreme, if we turned away every patch that only offered a 3% speed improvement or 4% size reduction we'd have an extremely slow and stagnated kernel in my opinion. Those 3 and 4 percent gains add up pretty fast.

      WRT Proof: http://forum.xda-developers.com/gala...-2014-t2427087
      I don't like bragging or boasting, the only reason I brought up the numbers there was because it was relevant to my case. There are even builds that have accumulated nearly 2000 downloads, however as of late the download numbers have dropped as the amount of time I've had for FML has dropped too, but I don't care, I actually have a good attitude towards it: "If they aren't happy with FML, I would prefer they give other ROMs a shot until they find something they enjoy, and/or give me some feedback as to what I could improve upon in FML."

      The OpenSSL bug was with code, it did pop into my mind though.

      Comment


      • #48
        Originally posted by Brane215 View Post
        Look at OpenSSL and heatbleed bug. How many people were using it ? For how many years ?
        Should we conclude from that that we shouldn't use (or have used) OpenSSL?

        Comment


        • #49
          Originally posted by MWisBest View Post
          Well I suppose the issue is more that LTO isn't something meant for using everywhere on a system where things are updated and changed constantly.
          Nope. That's exactly the environment that it should do best in - where there are many small things getting compiled into many bigger and those bigger being recomplied now and then. Since LTO can see inside at least fat binaries, it should be able to do its magic.

          For a case like that it's probably best to use it for individual things rather than on big dependencies. With something like Android everything is just compiled all at once and isn't updated in pieces etc.
          My point was that if this things crop up in such cases, there are serious bugs and inconsistencies in implementation and at least for me, solution is not in using LTO in a way that I simply won't see them.
          For my own playing that _might_ be acceptable, but not for something I intend to put in public use or give to customers.

          For the kernel though (what this thread was concerning) I don't see why there's any reason not to support LTO. Even if the gains aren't extreme, if we turned away every patch that only offered a 3% speed improvement or 4% size reduction we'd have an extremely slow and stagnated kernel in my opinion. Those 3 and 4 percent gains add up pretty fast.
          And WEEKS of lost time, mopping the cr*p afteer each such gain have lost me half of lifetime already. And it is adding up infinitely faster.
          Doing it for testing is fine, but using this in kernel for 1% gain on some irellevant test for 99.999% of the planet while risking for the thing to ge berserk and make fruit salad out of one's disk/s/ is IMHO moronic.

          Comment


          • #50
            Few more things, worth pointing out:

            1. LTO was experimented with before gcc-4.6, it was introduced with 4.6 and by first 4.7 it was declared as pretty complete, practically ready.

            From IIRC 4.6.3 to 4.8.2 I compiled each now revison of gcc and binutils in hope that things will get better. I can't remember one problem that I have solved with this. Even those _few_ packages that I managed to recompile with LTO have bitten me in the ar*e later.

            2. Have you noticed that even now even here no one actually knows even loosely WTF this is and how it is supposed to work ? It is all more or less anecdotal, at least out of the circles of compiler developers.
            Same with flag and their effects. I was told first that LDFLAGS have to contain CFLAGS, so that linker uses same optimisations as compiler did for optimal end effect.
            Then someone came along and told me this is totally wrong since one should NOT pass compiler flags to linker and that linker will get the info about the flags used from the object files themselves.

            Now hovewer is another day and I am told here that this is wrong and first version was correct all along.
            And tommorow is another day, so who knows ?

            Sometimes I get the feeling that for this crowd journey (= fiddling with compiling) is more important than the journey (= getting the code of desired quality).

            Comment


            • #51
              Originally posted by caligula View Post
              I also use Windows 8.1 for gaming and Mac OS X and can sure tell you that the rest of the world doesn't care. Mac apps are huge. Windows games are huge. The binaries are huge. Nothing is optimized well. Windows updater for Java doesn't delete old versions and the C: drive can be filled with 50 versions of Java 5 and 60 versions of Java 6. Nobody cares. On top of that they constantly run antivirus software which slows down the machine.
              We care. One might ask why you're on a Linux forum if your philosophy is so different.

              Comment


              • #52
                Originally posted by Brane215 View Post
                I've used freshest gcc I could find. Last few gcc bumps in gentoo were mine, since I couldn't wait for an official ebuild. I recompiled everything so many times.

                And even with gcc-4.8.2 and freshest binutils, my list of problematic packages was quite long and not shrinking much.
                I think you posted the list in another thread. But have you posted it to GCC bugzilla?

                Comment


                • #53
                  Originally posted by curaga View Post
                  I think you posted the list in another thread. But have you posted it to GCC bugzilla?
                  No. AFAIK they need easily testable small example that exhibits the error. This means that I was expected to sit down, understand what's going on and reduce bazillion files to one of a few lines.

                  I didn't have the time nor knowledge for such things.

                  And have I just showed them whole package in the report, what good would that do ? They must have known things aren't 100% peachy since so many packages failed.
                  And without doing the journey myself I'd just report what they already know - not all packages compile with LTO. And list of those was not shrinking, so they obviously haven't been investing work into this directly and one or a few more package names wouldn't mean a thing.

                  Comment


                  • #54
                    Having such a list publicly documented would help a lot of people: users to avoid using LTO on those, devs to focus on those. You yourself complained about lack of documentation on LTO.

                    If not a bug, then a gcc wiki page "Problematic packages with LTO" or such.

                    Comment


                    • #55
                      Originally posted by curaga View Post
                      Having such a list publicly documented would help a lot of people: users to avoid using LTO on those, devs to focus on those. You yourself complained about lack of documentation on LTO.

                      If not a bug, then a gcc wiki page "Problematic packages with LTO" or such.
                      maybe, but I have a strong feeling they know about those holes.

                      Besides, this has made me reGoogle sh*t about LTO.

                      So it sent me back to "GCC Internals Manual", which I either didn't see before or it was too high for me and my brain perceived it as white noise.

                      Be it as it may, now I can kind of follow it. And it deals with LTO and -fwhopr etc.

                      I'll try to read through it and see if I can contibute in more useful way after that.

                      Reading that stuff gives me headaches, but it sure beats long nights guessing my way around without any documentation...

                      Comment


                      • #56
                        Originally posted by curaga View Post
                        I care. One might ask why you're on a Linux forum if your philosophy is so different from mine.
                        FTFY.
                        One can care about linux and still not care about what you specifically care.

                        Comment


                        • #57
                          Originally posted by erendorn View Post
                          FTFY.
                          One can care about linux and still not care about what you specifically care.
                          I use Linux because it's great for servers, embedded and development desktops, but most games only work on Windows and Macs are great when travelling since their power consumption is low (only on Mac OS X), great UI/UX, great looks, good quality, nice.

                          Comment


                          • #58
                            Originally posted by ryao View Post
                            It turns out this is not strictly correct. Older Linux kernels use the ts bit on x86 to try to avoid reloads as much as possible:

                            http://stackoverflow.com/a/431323

                            It is still basically what I described earlier though. The kernel simply does not need floating point arithmetic. Sometimes, vector instructions are useful, but in those times, we use critical sections. Anyway, few kernel developers ever use these functions.
                            yes, floating point is banned from the kernel for a couple reason
                            one being just that, that its behavior is dependent on some flags (and they are cpu wide)

                            check out http://softpixel.com/~cwright/programming/simd/sse.php , the mxcsr register is for sse (x87 has even more stuff)

                            Comment


                            • #59
                              Originally posted by gens View Post
                              yes, floating point is banned from the kernel for a couple reason
                              one being just that, that its behavior is dependent on some flags (and they are cpu wide)

                              check out http://softpixel.com/~cwright/programming/simd/sse.php , the mxcsr register is for sse (x87 has even more stuff)
                              can't edit
                              http://wiki.osdev.org/SSE better explains what needs to be done
                              also i'm not 100% about how userspace floating point is handled, never had problems with it (tbh never needed to change the state, so..)

                              Comment


                              • #60
                                Originally posted by Brane215 View Post
                                Have you noticed that even now even here no one actually knows even loosely WTF this is and how it is supposed to work ?
                                Speak for yourself.

                                Comment

                                Working...
                                X