"CC_OPTIMIZE_FOR_PERFORMANCE_O3" Performance Tunable Dropped In Linux 6.0

[coder]

THAT'S what -O3 has always meant.

There's been a dearth of evidence presented to support this characterization. Not that it hasn't had bugs, but bugs have affected other optimization levels and beyond that, -O3 isn't fundamentally unsafe.

Sometimes you get lucky, but sometimes you don't,

A compiler is deterministic. The same set of inputs will produce the same outputs. When the inputs vary, the outputs will vary in often complex but ultimately predictable ways.

In contrast, a race car is fundamentally stochastic. You can operate within its design parameters and take the exact same corner at the exact same speed 99 times, only to have a part break and the car spin out on the 100th. Or maybe a gust of tail wind kicks up and it loses too much downforce. There's a fundamental degree of unpredictability that you can never get to 0.

It's fine for people to claim that O3 doesn't randomly explode any more, but that's basically just survival bias, always has been, and would still be even if the O3 problems are only half as common as they used to be.

That Clear Linux uses it is evidence enough to say that it's stable and usable.

It's not like Linus is going out of his way to make it even *nontrivial* to build -O3, let alone impossible.

He actually can't, other that to put something cheesy in the build scripts that actively suppresses it.

O3 is a second-class case for good reason,

Citation needed.

seeing non-developer forum warriors try to position themselves as somehow better informed than kernel devs is just cringe-inducing. Basement-dwellers struggling with inferiority complexes and trying to imagine they have some sort of secret knowledge that the sheeple lack and a mythical enemy is trying to suppress is the stuff of cults, not software development.

So much talk of cults, but the only ones I see being deified are kernel devs. They're not really so different than the rest of us.

Somehow, these "experts" insistent on O3 being the default are the same ones who find having to use KCFLAGS too much of a hurdle. The mind boggles...

Please name a single poster, in this or the prior thread, who actually advocated that it should be the default. I don't recall seeing even one.

A lot of the argumentation in these threads is between panicked users happy with the status quo, paranoid of some imaginary O3 mafia that's going to break their preferred distro's nice, stable kernels. Try actually listening to what people are saying, rather than lashing out against a some nonexistent collective as a foil for your fears and apprehensions.

from the perspective of "put up or shut up", since they've repeatedly failed to failed to manage the first part we'd all be a lot better off if they at least adopted the second part.

I'd suggest you don't have to read what they say, but it seems you're already taking that approach.

Furthermore, I'm skeptical of anyone who tries to silence a perceived minority opinion with which they disagree.

[Hafting]

Like anything, someone has to take the 1st steps to use -O3 a LOT. Beat the hell out of O3 from all angles. Find those pesky corner cases. .

Yes, this is necessary for improving the compiler. -O3 stuff that helps and is solid, eventually makes it to -O2 where it is good for everyone. So someone, preferably many, should experiment with O3. But the kernel seems to be the wrong item for such experiments.

If your application - small or large, crashes due to some odd -O3 buggy corner case, then you can debug that and report back to the compiler guys. It is some work, but not that hard. If your kernel crashes, your machine usually freeze and it is much harder to figure out what happened and where. But by all means, if you're good at pinpointing kernel crashes, use -O3 on the kernel.

[arQon]

There's been a dearth of evidence presented to support this characterization.

You... what? There's been *decades* of evidence for it: not in the thousands of anecdotes, but in the compiler errata. Which you *read*, ffs!

> Not that it hasn't had bugs, but bugs have affected other optimization levels and beyond that, -O3 isn't fundamentally unsafe.

Agreed, *any* nontrivial optimization has the potential to introduce bugs, and yes, that absolutely does include O2. I don't have a problem with that position. I can even agree with "isn't fundamentally unsafe", in that *conceptually* it isn't - that is, that what it does "should" work, 100% of the time, with all the inaccurate, explodey, or otherwise *known* potentially-broken stuff now living in Ofast/etc instead.

What I can't wrap my head around though is the argument that *in practice* O3 is somehow no more likely to cause problems than O2, because it simply *is*. Even without seeing it in the errata, it should be not just inferable but obvious: the more potential transformations you put the code through, the larger the potential failure surface. Shit happens. Even if a specific optimization is "provably safe" from a mathematical point of view, there's still the implementation to deal with, and the implementation can be very tricky and very prone to failure in unexpected ways.
I do mean just *that* set of O3 problems, not the cases where it generates slower code (or destroys the cache so badly that is has that net effect), which is a separate issue that I think is vanishingly rare these days for common use.

If you want to argue that "O3 is safe *now*", that's a different conversation and I'm happy to listen to it (though it's going to be a very hard sell). Claiming that there isn't more than enough evidence that it historically *hasn't* been though is Alternate Reality stuff.

> That Clear Linux uses it is evidence enough to say that it's stable and usable.

No, it isn't. Maybe that's the piece we're missing on.
You know the black sheep joke, right?

What Clear *is* evidence for is that, today, on at least Intel x64 - and possibly thanks to the efforts of the company that makes the CPUs and was willing to commit a team of developers and the associated infrastructure to it for years, but I don't follow the details - O3 is stable and usable enough to outweigh the potential risk of it in at least most cases.

That's what *all* this is, including your O0 suggestion to someone the other day: judgement calls and balancing risk vs reward. I'm not saying no code can ever be safe with O3: I'm saying that there is a very long history showing that the chances of *all* of a project being safe with it are slim at best, and - the validity of your point re O2 notwithstanding - that chance is noticeably lower than it is with O2.

If it helps to look at it from the other direction: if that wasn't the case, not only would there be a lot more code (almost certainly including the kernel itself) using O3, but O2 would effectively not exist, in the same way that O1 doesn't. I mean, think about it: when was the last time you saw something that used O1?

> Citation needed.

If you want to be like that about it, sure: how about the kernel itself?

The attitude I'm objecting to is the idea that everyone was somehow unaware that O3 existed. The adoption of O2 was overwhelmingly empirical: it's what's projects changed to *because* O3 kept exploding on them. Nobody ever started their first project and said "You know, I'd really like this code to run *kinda* fast, but I want it to be slower than it could be, just for shits and giggles".

That *is* where we ended up over time, because after the first, or second, or third project where you lost a week to tracking down just WTF the latest experimental optimization did to working code, O2 became the only sensible choice to make for the next project. Again, if you want to argue that that no longer applies, which you do, go right ahead. But to pretend that the sole reason that an entire generation of experts - working on machines literally hundreds of times slower than modern hardware, where we had to use assembler just to be able to reliably read 9600 baud serial lines - avoided O3 was just because we didn't have someone enlightened to show us the way, is comically arrogant.

> So much talk of cults, but the only ones I see being deified are kernel devs.

If anything I wrote even suggested that, it certainly wasn't intentional. What those more experienced than most do have though, is more experience, and dismissing that is at least as wrong as over-indulging it.

> Please name a single poster, in this or the prior thread, who actually advocated that it should be the default. I don't recall seeing even one.

I'm certainly not going to go searching through posts, so, sure: I'll concede that for the sake of argument. But if that *isn't* the goal, then why should O3 get any special treatment in the first place?

> A lot of the argumentation in these threads is

from people with no real understanding of the issues *at all*, on both sides. That being the case, I'd say that - from that group - those who choose to defer to the consensus of the best-informed are absolutely right to do so, whereas those who choose to attack that consensus without offering any evidence or reasoning should at best be ignored.

> Try actually listening to what people are saying

The comment I replied to, re cornering, is a perfect example of *why* the naive "XYZ uses O3 and AFAIK that's fine, therefore O3 is always fine" mentality is wrong.

I probably did then vent a bit more than I should have, but that's because I was tired of this thread the *second* time it happened, and we're now on what feels like about the sixth - including an entire article about testing O3 specifically. That frustration is *from* being subjected to too much noise on this topic.
But, yeah, you're right: much like O3, there are times when my output is not as good as it should be, and occasionally even times when it's just plain wrong. :P

> rather than lashing out against a some nonexistent collective as a foil for your fears and apprehensions.

Wow. I dunno what you're smoking, but "fears"? Really?
I thought I'd made my point, but apparently not, so I'll try again: I'm not "afraid" or "apprehensive" that something as esoteric as rebuilding the kernel in a nonstandard way is infinitesimally easier or harder. What I object to, based on a career spent entirely in developing high-performance and/or highly performance-sensitive systems, is the ridiculous idea that anyone advocating for anything other than O3 is somehow doing so out of ignorance of its existence, or some imaginary misunderstanding of the fundamental problems that optimizing compilers have had for as long as they've existed.
I also, since you've added it to the mix for god only knows what reason, object to the equally ridiculous idea that "fear" of anything other than Shit Not Working Properly is the concern here.

You're trying to overturn *decades* of poor experiences by... shouting loudly and attacking anyone who disagrees with you. *That's* what's annoying me right now. Nobody *owes* you a defense of O2: if you want the world to switch to O3, it's on *you* to show that it has value and is safe. We already tried and found it wasn't. And then did it again, with the same result; and again; and again. If you don't like that, then step up and do it yourself, and keep doing it until either you get tired of it too or you show that things really have changed, but you're not going to convince anyone by just being an asshole about it.

I know you can't prove that O3 is "reliably" safe, since that's essentially trying to prove the negative that it never generates *broken* code (potentially including performance-harmful constructs, but that's unfairly situational without additional factors and probably best left as "Part 2"), but there are things you could have done to support your position and didn't: namely, to point at the errata for the current compiler revs and say "Look! GCCn has gone a whole year without an O3-specific bug!". That would still have only been the first step in a long journey, but it would at least have given it *some* weight. A single anecdote has none at all, even over the timeframe you're talking about, because all that offers is the absence of evidence, whereas every O3 failure in that timeframe is yet another concrete case *of* O3 falling over.

As I've mentioned in one of the earlier threads on this thoroughly-flogged topic, I had a system that we built for six different platforms, and *one* of them triggered a compiler bug in *one* of the dozen third-party dependencies that meant we had to change the optimization level on it from the default. I'm not saying that makes the library authors incompetent, but I *am* saying that superficial testing is nearly worthless, superficial coverage is equally so, and that coverage has temporal aspects as well, not just LoC.

O3 has dug itself into a very deep hole, consistently, for years on end. Even if that *has* finally changed, expecting anyone to be convinced of that in a fraction of that time is wildly optimistic at best, and throwing insults at people for what amounts to simply having more experience in this field than you do isn't going to speed that process up at all.

> I'd suggest you don't have to read what they say, but it seems you're already taking that approach.


I did actually avoid this thread, but then I saw that it had somehow managed to pass 40+ comments, which if it had stayed on topic it really didn't have any reason to. I very much agree that reading it then instead of going to bed was a mistake.

The funny part is, the kernel is probably one of the least interesting pieces of code ever written on this front, and the nominal topic of this article in particular is even less worthy of attention.

> Furthermore, I'm skeptical of anyone who tries to silence a perceived minority opinion with which they disagree.

Huh. That's interesting, because I'm confident that the group clamoring for O3 at the start of all this was a pretty clear majority here. It wasn't until after Michael posted his benchmarks that that group mostly fell silent, and TBH I was pleasantly surprised by how many of them *didn't* then just try to change the topic to -march, PGO, LTO, and so on instead.
I expect this is one of those biases where whatever the "other" side is, that's the majority, because it makes us feel like heroic Davids.

Sorry for the epic post, but trying to explain nuance is like that.

[dekernel]

Just because I point out a contradiction in your position, that's no reason to sling insults.
I guess it's easier to blame me than reevaluate your own rationale.

Maybe I should not have used that word, but your statement was foolish. Nowhere did I say that optimizations should not be used. What I am saying that history has shown (at least twice as far as I know) that -O3 can be unsafe for the kernel, and history has shown that -O2 is safe.

It's well-established that -O3 is safe, and yet it seemingly remains taboo in the kernel community. I'll bet many of them are a lot more ignorant of it than you presume.
We're talking about GCC, though. For some other given compiler, maybe even -O2 isn't safe.

It is well-established to be safe for certain environment and certain code bases. Again, this is not the first time the kernel crew gave -O3 a try, but in each case, it has not worked out well due to edge case scenarios.

Why are you worried about what I'm worrying about? Asserting your own right to have a position on the matter while denying mine is hardly fair.
Kernel performance is relevant to my interests. So, it's a concern to me when compiler optimizations aren't being explored and exploited to their potential.

I'm worried because I want my kernel to be 100% reliable all the time....every time, and I fear that the desire for the small percentage increase at the cost of potential instability is foolish for my situation and I bet many others.

That's a very arbitrary judgment. Sure, Linux is usable by a lot of people, for a lot of things. So, in some sense, there's a notion of competency. Still, to say they're doing "a really good job" seems like it should somehow account for the amount of engineering hours and testing resources it's taken to achieve its current level of quality. And that quality metric really ought to be based on the normalized defect rate, rather than some vague sense of "this thing works for me".

I would agree 100% with you on this, but personally I just don't have the time to work through this metric so the best I can use is my actual runtime experience to gauge the "really good job". To date, I have zero crashes on either my workstation or production servers due to a kernel bug so that is the best yardstick I have to date. I can also say this in regards to other engineers I know that use Linux for their production environments. I do realize that is not a very quantitative statement, but again, it is best I have based upon my experience as well as a few others, and since I have to use something as a basis for the decision...

Furthermore, Phoronix publishes an endless series of news stories about poorly-conceived datastructures and algorithms having to be ripped out replaced, not to mention all the performance bottlenecks that Jens Axboe is tripping over, left and right. So, I think there's room to make an argument that the kernel devs haven't overall had performance as their main focus.

Finally, I don't really have an overall sense of the distribution of opinions about compiler optimizations, among the kernel development community, do you? I think we can say there's not an overwhelming push for > -O2, or else Linus might have caved. Beyond that, I wonder if you're conjuring a broader consensus than what truly exists.

Well, since there is a push for -O3 there must be a group that feel it is a good idea, but beyond, I have no concrete numbers. What I don't want is a small but quite vocal group push this through without a consensus amongst the Colonel's and above, and since each push has failed, I am guessing there is more in the -O2 camp versus the -O3 camp at this point.

Please make sure to understand that I am staying solely focused on the kernel here when it comes to -O3. When people say that -O3 is safe without any type of qualifiers tells me those people are focused far too much on their own project/environment which typically runs in a very controlled environment where the kernel is the exact opposite of that. Yes, proving that -O3 is safe for the kernel is like proving a negative which is logically impossible, but to date, -O3 has proven to be unsafe in enough circumstances that the owners of the code deem it unsafe which is more than good enough for me at this juncture in time.

[CochainComplex]

The way I imagine racing is that designers are dealing with a model of the car which they try to optimize in CAD, wind tunnels, etc. -- trying different approaches and trading off different variables to see what overall configuration works best. Once the thing is built, then the problem becomes more one of data science and dealing with a collection of simpler, more statistical-type models of the car's behavior which they develop and use to optimize specific parameters for a given race track set of weather conditions, tire/fuel strategy, etc.

I have no idea how accurate that is, but when it comes to predicting how fast a car can take a given corner, the driver is going to get an intuitive feel for how the entire package is handling and can feel approximately where the limit lies. This part, you might call the original form of "deep learning".

not am expert but I think its how its done. I guess in Formula 1 the Drivers give there feelings and their wishes as feedback to the engineers. For the adjustment. Some drivers do prefer an oversteering or understeering car. Btw its quite worthwhile to watch youtube vids about the driving analysis of e.g. Michael Schumacher - who supposed to have a very sensitive feeling for the maximum of the car in each situation. https://www.youtube.com/watch?v=2Czacauviyk

[coder]

Yes, this is necessary for improving the compiler. -O3 stuff that helps and is solid, eventually makes it to -O2 where it is good for everyone.

Note that -O3 is more than just a staging ground for stuff to eventually move to -O2. It also contains optimizations that aren't always a win, or that are more costly in compiler time or code size.

So someone, preferably many, should experiment with O3. But the kernel seems to be the wrong item for such experiments.

You seem to be presuming a level of risk that has yet to be demonstrated, if it even exists.

I understand risk aversion. What I don't understand is this seeming aversion to knowledge, so that people never have to accept the possible reality that -O3 isn't the risk they thought it was.

It's like... in the age of data science, we're still being ruled by our lizard brains and making big, important decisions based on lore, anecdote, fear of the unknown, and fear of knowing.

[coder]

not am expert but I think its how its done. I guess in Formula 1 the Drivers give there feelings and their wishes as feedback to the engineers. For the adjustment.

The cars have hundreds of sensors, these days. No doubt, the driver's feedback is a key part of it, but they're also looking at a massive amounts of data.

[arQon]

Well, since there is a push for -O3 there must be a group that feel it is a good idea, but beyond, I have no concrete numbers.

Inferences are as good as we're going to get: Linux is not a democracy, which is one the reasons why bad *new* ideas tend to not get too far. (Though I'd bet simply not having Sales, Marketing, or UX reduces that by at least a thousandfold). :P

> What I don't want is a small but quite vocal group push this through without a consensus amongst the Colonel's and above, and since each push has failed, I am guessing there is more in the -O2 camp versus the -O3 camp at this point.

Well, "small" is another inference, but it also needs to have the "democracy" aspect attached. Anyone *can* be a maintainer, but they're generally more experienced than other contributors, for obvious reasons. By the time you get to the Lieutenants, they're either MUCH more experienced or have "outperformed" their peers for years, which I'd call a wash for the experience. Given that O3 has failed multiple times on the kernel, at least all of that that group will be aware of it.
IOW, among the people who make the decisions, there is likely to be, if not almost universal distrust of O3, at least a very strong majority that feels that way.

We can probably assume that at least one LT signed on to the config change: without at least that much support I don't see it ever making it into a release. As for how large that group was, I share the curiosity but it just doesn't really matter match *yet*.

WAG: one guy brought it up out of curiosity. He asked around his group, and the last O2 failure was "long enough" ago that enough of them through it was worth trying again, and offered the patch up the chain to their LT, who agreed.
I'm actually more curious as to why it got reverted. It can't just have been Michael's testing, but I'm not curious enough to wade through LKM tonight so...

[karolherbst]

karolherbst Can I borrow a box of popcorn? I ran out while reading this thread.

sure, here ya go: 🍿🍿🍿