Arch Linux Could Use Some Help With Toolchain Maintenance

[binarybanana]

Weird that they seem to have such a hard time. Bumping glibc isn't difficult if you can get it to build. That's pretty much it. Getting there is sometimes tricky, but other than that it'll be backwards compatible with everything previously compiled. Bumping gcc is no different unless there are changes in how some things are linked, like libssp, libgcc, etc., statically or dynamically, or when default compiler options are changed like pie. Or changing target triplets from x86_64-unknown-linux-gnu to x86_64-arch-linux-gnu or whatever. Worst is usually just new warnings and stupid projects using Werror in release builds so things break for no reason... Which is why I patch Werror out completely nowadays. Only downgrading is a pain, but in emergencies stubbing out missing symbols tends to work. At least that's been my experience. If all else fails, just take a look at how Gentoo makes it compile with whatever settings Arch uses.

But it seems they don't even have a working toolchain that compiles their current glibc. If it's so borked that it can't compile with itself they might need to bootstrap from an earlier version or "cross-compile" from somewhere else.

I am using his alternative toolchain and the two biggest differences is that you need the outdated GCC in the main repos to build kernel modules(DKMS) as you need to build modules with the exact same compiler that was used to build the kernel

No, you don't need the exact same compiler. That hasn't been true for over a decade, back in the gcc2/3 days. Maybe a bit later, but not that much later. I haven't tried mixing gcc kernel with clang modules because my kernel is monolithic but with some luck even that might work. Probably not with LTO, though. Mixing kernel patch releases is more risky but with no configuration changes breakage is rare. I think it requires enabling some symbol versioning option in the modules section in menuconfig. Anyway, at least for different gcc versions it is not an issue, especially a minor version bump like 11.1 -> 11.2, which should be ABI compatible (or it's a bug AFAICT).

[kokoko3k]

Next time please try to understand what you are replying to before being condescending. I know the requirements and I'm not complaining about the existence of TUs or that it's hard to become one, I'm complaining about the fact that a regular user has no way of contributing at all(https://bbs.archlinux.org/viewtopic....270662&p=3 see post #60, a user's patches were ignored 5 months ago and the toolchain is still outdated).

And before you try to sugarcoat it, most regular users who are competent enough and willing to contribute to packaging are not willing to spend their even more of their time on establishing a presence in the arch dev scene and finding two TUs to sponsor them just so that their contributions that they created in their own free time get used in Arch. Most people won't (and they currently don't) even bother when getting people to at least look at or use your build scripts/PKGBUILDS is harder than writing the PKGBUILDs in the first place. Not exactly good for a project based on voluntary contributions.

You could have TUs AND git repositories for PKGBUILDS that regular users can send pull requests to which can be commiteed by TUs if they want to. This way TUs would still do what they do now and have the ability to approve user contributions if they wanted to. Everything that goes to the repositories would still have to be approved by TUs and users could contribute too, which would have prevented the current debacle. Restricting everything to a circle that users can't access just creates more work per maintainer and creates uncertainty.

I reply to what i read, next time please try to be more specific if you want others to understand, or just write [RANT] before.

That said, your thinking is valid as long as there is someone willing to get PR, analyze them and decide what to do ...which is exactly what we're talking about.
You read #60, but again, fail to understand what TUs are for. Maybe you missed #65?

[guglovich]

Valve, help!

[Nocifer]

Ouch

TL;DR: This is a problem that unfortunately affects the open source community as a whole, and we'd be best off acknowledging it sooner rather than later: the modern tech culture has undergone a paradigm shift, and while the "old guard" of open source is slowly but steadily stepping down, there's less and less younger people each day who are both interested in contributing to open source software on a volunteer basis AND are able to do so on a technical level, especially when important parts of the infrastructure are concerned (things like e.g. the compilation toolchain or security).

What I read in that forum thread is nothing other than proof for what I've already been seeing/feeling for the least couple of years or so. This is not the first time this has happened or the first Arch package that has been thus affected (e.g. I remember when IBus was kept out-of-date since June 2019 and with a hard dependency for Python 2, and even though users had provided the necessary patches, until it was finally updated in December 2020 - almost a year after Python 2's EOL; or the case of the lib32-gst-plugins-{bad,ugly} packages, which have been and still are on the to-do list to be properly packaged in the repos since 2016), but it's the first time this concerns such high-profile packages as gcc, glibc, binutils, et al - which in my mind (and of others, judging by the responses in that thread) it means this concerns Arch Linux as a distro.

Unfortunately, I think this is a more widespread problem around Arch Linux and its community, and IMHO it comes down to this: the old guard has grown literally old(er) and has mostly left Arch for other pastures, while at the same time there is simply not enough new blood available to replenish the talent that is now so severely lacking. And the problem is worse: I even remember being surprised at hearing underhanded comments from Arch devs against other Arch devs about "neglecting their community duties" due to "being busy with their side projects" while watching the Arch Conf presentations back in 2020 - that was my first real hint that something is really wrong behind the scenes, and it's no longer limited to isolated issues with package X or maintainer Y.

I had high hopes about the new Arch leadership, but unfortunately it seems that things are not going so well: Arch feels like a leaderless project nowadays (and that's not a jab against the current leader, who's AFAIAC done a very admirable attempt at trying to piece the broken pieces back together); this is evident pretty much everywhere, from the lack of quick updates to the lack of major modernization initiatives for which Arch used to be famous for (yeah, there's certainly talk about them, but that's not nearly enough) to the lack of communication on the part of the distro's core team towards the wider community. Even the glorious Arch Wiki is suffering from bit-rot in many of its articles, with sections or even whole pages containing outdated or flat out wrong information. Why? Because the people that made the Wiki glorious back in the day are mostly not around anymore, and the new people are either too lazy, too ignorant... or there simply aren't any new people around. So the info on many parts of the Wiki has stagnated, and thus is slowly but surely losing its relevance in the long term.

On my part, I believe this issue is more widespread and it doesn't concern Arch only, rather it's due to a cultural shift - we have to face the reality that unless something changes radically in the future, young people on the whole are no longer as interested in open source and volunteering and forming/participating in tech communities as they used to be just a scant generation ago (i.e. in the '00s), and this has or at least will very probably have a terrible effect on Linux as a whole (has anyone ever wondered what will happen to Linux - the kernel - in, say, 10 years when the current old-ish guard steps away?). It's fast becoming a situation where open source is dependent on corporations to provide the effort necessary to keep it afloat and up-to-date in terms of even such basic things as security fixes. It's not a secret that even now, many if not most of the new and shiny Linux features (from Wayland, to Gnome and KDE, to device drivers, to the kernel itself) are fueled or at least majorly assisted by some corporation or another that have invested themselves in that feature and payed some devs to implement it and/or maintain it. And don't even get me started on what will happen if and when Linux becomes even a tiny bit bigger than it is now - say 5% of the market - and the requirements in manpower for it to stay afloat increase proportionally.

In the case of Arch I believe the situation can and will be resolved in either of three ways:

1) The old guard returns (small chance because Life™)
2) The younger generation develops a greater interest, along with the required technical merit - that's a toughie-, in stepping up and taking the mantle and filling in the crucial open source dev roles that have been left empty by the old guard.
3) Some corporation like Valve steps in and provides the necessary manpower by replacing volunteers with paid developers - essentially either adopting Arch, or forking it into a new corporate-backed distro which will eventually replace or at least majorly overshadow Arch - think Proton and Wine, but on a larger scale.

My hope is for #2, but (unless something major changes) my bet is unfortunately on #3. At least we're lucky it's Valve and not e.g. Microsoft.

And before you try to sugarcoat it, most regular users who are competent enough and willing to contribute to packaging are not willing to spend their even more of their time on establishing a presence in the arch dev scene and finding two TUs to sponsor them just so that their contributions that they created in their own free time get used in Arch.

Oh my "${spiritual_entity}", this. So much this. Everything else is par for the course, but having to go out of your way to find two strangers, establish a friendly relationship with them and convince them to trust you and vouch for you? OK, I'll admit it's not that hard, at least not if participating in such an online community is a major hobby of yours. But this is 2022, not 2002 or even 2012, and when there's hundreds of online communities out there that vie for your attention and participation every single minute, not to mention that little thing called Life™ which kind of grows disproportionately and becomes this all-encompassing and all-consuming monster after the age of say 25-30... why would you freaking bother going to all that trouble just for the sake of having the "privilege" to share a few of your patches now and then?

/rant

[leo_sk]

If by "help" you mean "wait until 2-3 weeks after Arch has done the work, then DDOS part of the Arch infra" then yes, they can help.

If you look at the bug, it was not something they could predict without knowledge of infrastructure available at AUR. And it was resolved pretty quickly when an arch dev submitted it to gitlab. You people act like arch itself never did anything wrong in first 5 years of existence

[ms178]

Why is there a segfault in Chrome? I don't know.
Why did you add those extra compiler flags? I don't know.

Ah, success! I found the motivation to try Allen's PKGBUILD and found the culprit as it seems Glibc doesn't like

fno-semantic-interposition

and

Bsymbolic-functions

(or

Wl,--as-needed

). A minimalistic set of flags did the job finally!

On the other hand, the limitations of Allen's toolchain are quite severe, without multilib support there are no custom Wine/Proton builds and I also found a LTO-related bug which I don't have with my own toolchain (which is from End of January). As I used his pre-built packages to build Glibc, I'll try to build the other packages myself, also my Endevour-OS is already quite a Frankenstein-build.

[CochainComplex]

Thanks for this news piece, the situation is less than ideal. Such crucial packages need a good maintainer and as there are other packaging efforts underway downstream, I wonder why that is not a focus for these projects to invest their time to optimize it, too. Thanks to the dire state I had to learn how to build a profiled GCC myself (thank you Clear Linux for some inspiration for optimizations!), but Glibc is a beast, I get segfaults in crucial programs like Chrome or symbol errors and never succeeded there. Maybe I hit some bugs or problems with the PKGBUILDS, also the build order is important... but that was a rather frustrating experience as it takes down the whole system and cannot be saved from a chroot environment.

Update: Reading through the Arch-thread with some more glory details, the technical details for solving these problems are over my head - especially dealing with the build failures. Also Arch and all of the other downstream projects seem to be in a bad state if they cannot provide security updates any longer to this crucial part of the system. This screams for more automatation, automated building and testing. Was Valve aware of these issues when they made their decision to switch to Arch? Will they come to the rescue and invest the needed ressources here?!

Glibc is really a beast. I'm playing around with gentoo and I'm also using some flags mentioned in the clear Linux spec files for the majority of my gentoo builds. On top I try to compile world with aocc since I'm on an amd system. But glibc does not like too much of fiddling with flags or alternative compilers. It almost always compiles fine but after that one might find sudden segfaults here and there.

I'm wondering why it compiles fine very often but then software breaks. Is it code quality or what is the reason?

[ihatemichael]

Time to switch to Gentoo.

[KaoDome]

A tad off-topic, but some posts in the thread have made me thing of this.

How come GCC, Glibc and pals are so finicky? Why is building and packaging them such a daunting task? You'd think mature, stablished projects would have a streamlined building process and somewhat deterministic build. There are plenty of tests for each right? Unit, integration, etc. I mean, I'm used to seeing check() functions in PKGBUILDs but the way things seem to be with these toolchain packages makes me wonder what gives.

By the way... with the myriad of distributions there are out there I would have thought a common build recipe or skeleton, for lack of a better word, would have been created leaving but a few choices to the package maintainers, like building flags or artifact placement.

[lano1106]

This article is a blessing. thx Michael to have looked into it and made it a story. Hopefully, having the spotlight on the issue will motivate the maintainers to increase the priority of updating the toolchain!