Profile Guided Optimizations (PGO) Likely Coming To Linux 5.14 For Clang

[sdack]

Your entire argument is based off assumptions, so do me a favor and follow your own advice.

No, this is just another assumption you are making. I am drawing from experience.

[mdedetrich]

The very idea to base a decision merely on assumptions is unprofessional. As is most of your comment.

Your entire argument is based off assumptions, so do me a favor and follow your own advice.

[sdack]

Its not unprofessional to assume ...

The very idea to base a decision merely on assumptions is unprofessional. As is most of your comment.

[mdedetrich]

While I appreciate the candour, do note that GCC has a much longer history. LLVM/Clang's history is a short one and it has only recently become able to compile the kernel. It is unprofessional to assume a new piece of software was better than one you have been using for decades, that enabled you to get to where you are today and where you have made plenty of experience with.

Its not unprofessional to assume that new software can be better than older software, in fact having such a view of things I would argue is primitive. Newer software can often be better than older software since it can start off with a better design and/or better technologies which have a better net effect then any "decades of bug fixing" (its often feasibly impossible for older software to significantly change fundamental design of said software).

This is btw why LLVM was started and why Apple poured so many resources into it, Apple used to use GCC but the design was so monolithic that they had technical issues with implementing what we see now as basic IDE/Editor inspection cabilities for C/C++/objective C code

The screw ups were not only on the compiler side, but the compiler helped in uncovering issues with the kernel, too. Nor is LLVM/Clang completely independent from GCC, but the two do share some cooperation. The problem is with the amount of expectations placed into compilers, and once LLVM/Clang bites you will it be just the same and you end up making a new experience, which you can hate, or appreciate and learn from.

Considering its young age, LLVM hasn't been any worse in the "screw up" department compared to GCC. You have to realize that LLVM is now used at a massive scale, so its having plenty of battle testing

In open source projects can one turn a problem into a win-win by learning from it and by cooperating with other projects, by passing on the knowledge, and allowing others to learn from it, too. Or you can do it wrong by passing around blame, give hate speeches, and what not. Not everybody understands this at start and it has resulted many times in all kinds of conflicts in the open source community.

I am going to leave it at that. I am looking forward to 5.14, 5.15 or perhaps 5.16, and maybe we see the support for GCC LTO finally landing within the mainline kernel.

Sure, but at the end of the day Linux is going to use the tool that is better for the job. In the case of LTO, that was Clang over GCC.

[sdack]

Thats probably because GCC has historically screwed up enough times that its been put on Linus's "observe with care" radar where as Clang's LLVM LTO has proven itself by being shipped in millions (billions?) of androids devices as well as being used in cloud providers, in other words its had plenty of testing.

While I appreciate the candour, do note that GCC has a much longer history. LLVM/Clang's history is a short one and it has only recently become able to compile the kernel. It is unprofessional to assume a new piece of software was better than one you have been using for decades, that enabled you to get to where you are today and where you have made plenty of experience with.

The screw ups were not only on the compiler side, but the compiler helped in uncovering issues with the kernel, too. Nor is LLVM/Clang completely independent from GCC, but the two do share some cooperation. The problem is with the amount of expectations placed into compilers, and once LLVM/Clang bites you will it be just the same and you end up making a new experience, which you can hate, or appreciate and learn from.

In open source projects can one turn a problem into a win-win by learning from it and by cooperating with other projects, by passing on the knowledge, and allowing others to learn from it, too. Or you can do it wrong by passing around blame, give hate speeches, and what not. Not everybody understands this at start and it has resulted many times in all kinds of conflicts in the open source community.

I am going to leave it at that. I am looking forward to 5.14, 5.15 or perhaps 5.16, and maybe we see the support for GCC LTO finally landing within the mainline kernel.

[mdedetrich]

That is actually not true. Andi Kleen wrote the LTO support for GCC all the way back to kernel 3.7. Torvalds just never wanted it in the kernel. It was then that Google picked it up, used it for LLVM/Clang and pushed it that we now have it in the mainline kernel. Andi Kleen keeps updating the patches for GCC LTO support but so far have these not made it into the kernel. There certainly is a bias here and Torvalds is known for his grudges against GCC.

Thats probably because GCC has historically screwed up enough times that its been put on Linus's "observe with care" radar where as Clang's LLVM LTO has proven itself by being shipped in millions (billions?) of androids devices as well as being used in cloud providers, in other words its had plenty of testing.

[sdack]

(*) I think that those initial patches came with certain caveats, like "experimental, don't use", and the data to support the claimed improvements were not there, so those and others were all contributory factors in the rejection, at that time.

The LLVM/Clang LTO support is marked as experimental. And proof was there in form of benchmarks and these showed quite some interesting gains. So did networking performance increase by 12% or so iirc. Hence my comment regarding network-oriented distros getting the most gain from it. Networking takes mostly place within kernel space and it is to be expected to gain the most from LTO as well as PGO.

Torvalds then took up parts of the GCC LTO patches. However not in favour of GCC LTO, but because hidden issues within the kernel had surfaced from its implementation. So parts of the patches were taken up as regular fixes.

[CommunityMember]

That is actually not true. Andi Kleen wrote the LTO support for GCC all the way back to kernel 3.7. Torvalds just never wanted it in the kernel.

Well, it was (as it usually is) slightly more complicated, as Linus stated at that time he did not believe LTO had proven it's mettle yet(*). Google showed sometime later (at least with LLVM) that the improvements were there, and they deployed it on many millions of devices in the real world. Proof by existence/benefits tends to be a strong factor in accepting patches.

knowing that Torvalds hates GCC

I don't think he hates(**) GCC, as much as the kernel community has been burned more than a few times on interesting code generation, and avoiding finding of new ones is not an unreasonable way forward (fool me once, shame on you, fool me twice, shame on me).


(*) I think that those initial patches came with certain caveats, like "experimental, don't use", and the data to support the claimed improvements were not there, so those and others were all contributory factors in the rejection, at that time.

(**) Hate is often more of an emotional reaction. Dislike, or despise, tends to come with specific reasons.

[sdack]

Those that believe in meritocracy would believe that the best compiler should win, regardless of the whether it is GCC or LLVM (or something else). Right now, it is clear LLVM is where the action is, and the best results are. Nothing stops you from improving the kernel (and GCC) to make sure it remain a viable choice. I would encourage you to do so (there have been patches floating around for quite some time for GCC LTO of the kernel, so that might be a place for you to start).

That is actually not true. Andi Kleen wrote the LTO support for GCC all the way back to kernel 3.7. Torvalds just never wanted it in the kernel. It was then that Google picked it up, used it for LLVM/Clang and pushed it that we now have it in the mainline kernel. Andi Kleen keeps updating the patches for GCC LTO support but so far have these not made it into the kernel. There certainly is a bias here and Torvalds is known for his grudges against GCC.

I will simply compile the kernel with Clang, knowing that Torvalds hates GCC. It is simply the smarter choice to use Clang as I do not want to get caught up in the fight. I am however aware of the nonsense and choose GCC for most other things, unless again I see a good reason not to.

As you say, meritocracy should always win, but sometimes it does not and it takes a little time.

[CommunityMember]

This is just another marketing attempt at pushing Clang to become the default compiler for all distros instead of GCC because they hate GNU and what it stands for.

Those that believe in meritocracy would believe that the best compiler should win, regardless of the whether it is GCC or LLVM (or something else). Right now, it is clear LLVM is where the action is, and the best results are. Nothing stops you from improving the kernel (and GCC) to make sure it remain a viable choice. I would encourage you to do so (there have been patches floating around for quite some time for GCC LTO of the kernel, so that might be a place for you to start).