Fedora 30 To Take Stab At Eliminating Excessive Linking

[Weasel]

Doesn't work so fine if you want to build static and public variants of the lib. Then you need macros and ifbutting. And if you place that pragma before any imported symbol your code will misbehave.
IMHO this shouldn't end up in code at all, but be defined solely by some linker-export-map (would need the linker to be able to patch in local or shared function calls)

Not if it was the default. The only reason you need "pragmas" is because the default is stupid at the core.

Of course you'll need some attribute or pragma to export properly. If ELF was designed like PE, you'd just need dllexport attribute, and that's it. A simple macro that turns it into either dllexport or dllimport, depending on whether you export it or import it. Not a problem at all.

Yes, and if you want to be able to provide your app that interface then you depend on the app. I'm not arguing against using '--as-needed', but just as compiling with LTO this will break stuff that previously worked. A little nuisance if its your own code, enabling it on a distro-level without testing the packages can cause that trouble.

Conversely if you define a proper interface you should not run into problems if both ends hold up to the specification. It should not matter where the implementation comes from, in that respect - taking out the linker-machinery of resolving DSOs and making the App responsible for dependencies of the plugins is a good idea if your plugins ought to isolated code (with expensive verification on each modification).

First: your app should absolutely depend on the other app. That's just common logic: if it's a plugin for app X, then it *must* depend on app X. What kind of nonsense is there to be a plugin for something and not depend on it explicitly?

Secondly, I'm not disagreeing with you that it might break stuff. I'm very well aware it will break stuff. The thing is that it should not have been a thing in the first place. I'm not blaming --as-needed for breaking stuff, even though it probably does and you're right. I'm blaming the pathetic code that breaks with it for being designed like a potato.

LTO is similar, but unfortunately it's full of bugs. If it was perfect, then no project should ever fail to compile with LTO: if it does, it's the project that's designed badly, not the LTO concept.

the only difference is that a linked dependency will be opened with RTLD_GLOBAL, that means the symbols of the DSO will be put into the global namespace. ODR is a term from C/C++ and does not apply to the linker (it does not care or define anything at binary level), there are no inherent conflicts or issues if you load multiple DSOs with the same public symbols. You should know what you are doing though.

I'm saying that the "global namespace" should not be a thing. It is the worst idea ever.

Maybe explain exactly what ELF (or rather SYSV which defines the linker behavior AFAIR) does wrong in comparison of COFF, cause that handwaving with unfit programming terms on a topic about the dynamic loaders symbol namespace gets you nowhere?

Global namespace. It's as retarded as one-letter global variables in code.

I usually don't argue about this, but I'm personally sick to death of people (not you!) encountering so many "clashes" and "issues" with ELF (if you knew about the other flatpak threads here) due to this idiotic idea of a "global namespace" that shouldn't have ever passed the quality assurance.

And then they have the audacity to think it's an universal problem, and claim COFF/PE/Windows (or Mac) suffer from it as well, when they don't because they have no such garbage. It's only ELF that suffers from it, but they still insist on their bs, so it triggers me a lot. (btw it's not just Windows, since Wine run DLLs just fine and don't suffer from it, it's also used in UEFI so...)


You probably ask what has that do to with --as-needed?

Simple: if you need to import a symbol, you must be forced to specify which module it is from. ELF doesn't force that, because of the "global namespace". Ergo this whole issue exists because of the "global namespace" being a thing. If you must specify which module it is from, by the format of the module (COFF), then it is literally impossible to ever have an application fail with --as-needed, in fact --as-needed would be the implied default.


tl;dr Whoever had the idea to import symbols *without* specifying which module they are supposed to be from should be shot.

[discordian]

Not if it was the default. The only reason you need "pragmas" is because the default is stupid at the core.

Of course you'll need some attribute or pragma to export properly. If ELF was designed like PE, you'd just need dllexport attribute, and that's it. A simple macro that turns it into either dllexport or dllimport, depending on whether you export it or import it. Not a problem at all.

Agreed, bit I dont find pragmas to be the ideal solution - its akward to use conditional directives for compilation (import/export or static library) when this is a purely linker-driven decision.
You typically already have a file for symbols and versioning, would be easiest to either directly feed that to the compiler or even better let the linker fill out the blanks correctly.

First: your app should absolutely depend on the other app. That's just common logic: if it's a plugin for app X, then it *must* depend on app X. What kind of nonsense is there to be a plugin for something and not depend on it explicitly?

you cant tell a plugin to use symbols from the app (which one if multiple are used). AFAIK thats neither possible with COFF.
And you might not want to use filenames as dependencies but interfaces if you use isolated environments, the app can then check if it fulfills the requirements. I don't want to discuss this further though, its tangential (not the normal system library dependency scheme).
using ELF with customized linking/loading saved me alot work creating and maintaining tables.

Secondly, I'm not disagreeing with you that it might break stuff. I'm very well aware it will break stuff. The thing is that it should not have been a thing in the first place. I'm not blaming --as-needed for breaking stuff, even though it probably does and you're right. I'm blaming the pathetic code that breaks with it for being designed like a potato.

You further seem to blame ELF and SYSV when the toolchains would be more responsible.

LTO is similar, but unfortunately it's full of bugs. If it was perfect, then no project should ever fail to compile with LTO: if it does, it's the project that's designed badly, not the LTO concept.

LTO is quite troublesome as there are have been bugs in compiler/linker implementing it aswell.

I'm saying that the "global namespace" should not be a thing. It is the worst idea ever.

Global namespace. It's as retarded as one-letter global variables in code.

Extensions like gnu symbol versioning exist eg. "perror@GLIBC_2.2.5", that could be described as namespace. whether you iterate over loaded dlls (namespaces) then iterate symbols or iterate over global symbols with namespace information should not matter.

I usually don't argue about this, but I'm personally sick to death of people (not you!) encountering so many "clashes" and "issues" with ELF (if you knew about the other flatpak threads here) due to this idiotic idea of a "global namespace" that shouldn't have ever passed the quality assurance.

And then they have the audacity to think it's an universal problem, and claim COFF/PE/Windows (or Mac) suffer from it as well, when they don't because they have no such garbage. It's only ELF that suffers from it, but they still insist on their bs, so it triggers me a lot. (btw it's not just Windows, since Wine run DLLs just fine and don't suffer from it, it's also used in UEFI so...)

You also have the ability to replace symbols from DSOs. I dont know whether this offsets the traps those scheme brings, but several important tools for tracing and lightweight isolation would not be possible without (strace, various heap tracers, lttng, fakeroot, bear), nor replacing easily replacing the heap managment for ex with jmalloc.
The common theme and my opinion is, that these possibilities are great, but should be opt-in and I haven't found any use for them not purely involving functions from the basic runtime libraries (glibc, pthread).

I just havent encountered those issues you speak off in any relevant manner, but it seems to me no more a general design issue then initializing variables before you read them.
The "DLL Hell" is a known issue in windows, solved by pretty much any app using their preferred or even private version of DLLs, I would argue that these makes whatever issues you face pretty rare (and not because of technical superiority of COFF).

The thing I hate most about the default DSO exports is that eg gcc will create bad code, often with linker relocations for calling a function/variable from the same DSO. Thats even including platforms where there is no dynamic linker and you just want to build PIC.

You probably ask what has that do to with --as-needed?

Simple: if you need to import a symbol, you must be forced to specify which module it is from. ELF doesn't force that, because of the "global namespace". Ergo this whole issue exists because of the "global namespace" being a thing. If you must specify which module it is from, by the format of the module (COFF), then it is literally impossible to ever have an application fail with --as-needed, in fact --as-needed would be the implied default.

Yep, that would be clean engineering.

tl;dr Whoever had the idea to import symbols *without* specifying which module they are supposed to be from should be shot.

doesn't work with functions exported from the application (could further be multiple apps/names) that a plugin is supposed to use. Only valid for system libraries.

[F.Ultra]

Nah. See my previous post, it will hopefully answer.

I'm starting to "get" where you guys come from: the design-less land of crappy designed code that lacks proper interfaces and documentation. Internals must not be exposed, unless they're part of the interface. Relying on that is even worse. I mean, ffs, interfaces are important even within a single project. C++ even has "private" keyword for this exact reason, and that's within one project. And yet some people want everything exposed across libraries? wat?

Encapsulation is a thing. Public symbols are a nightmare: symbols should be tied to a specific library only and never ever clash outside of it.

If I am to use a C++ analogy, it would be: each symbols should be in a separate namespace, defined by the library, so:

namespace liba
{
symbol1;
symbol2;
}

Anything else is a spaghetti nightmare.

What else, let's make inter-function goto the default? (longjmp) It's the same fucking mentality! Let's expose all labels in functions (implicit by structured control flow constructs) and enable us to jump to them with goto (longjmp)!

Each to their own I guess, there are upsides to namespaces and there are downsides to namespaces, but that was not my point anyway. My point is that you should drop this emotional connection you have with ELF and realise that the on-disk format have exactly nothing to do with this. You can export only the exported symbols in ELF just fine and you can export every single internal symbol in PE just fine as well.

[F.Ultra]

doesn't work with functions exported from the application (could further be multiple apps/names) that a plugin is supposed to use. Only valid for system libraries.

Well to be honest the application could hand out a struct on say plugin_init() filled with the exported functions the plugins should be allowed to use, since that was how I was forced to make it work on Windows it became the Unix solution as well (better to have one single interface).

[discordian]

Well to be honest the application could hand out a struct on say plugin_init() filled with the exported functions the plugins should be allowed to use, since that was how I was forced to make it work on Windows it became the Unix solution as well (better to have one single interface).

Yeah, that has other drawbacks, as now every function call is now explicitly using this struct, if you want back/forwards compatibility then things get even more complicated.
With ELF symbols + versioning you have that solved and could decide at link-time which functions will get statically linked and which are from the app.

[Weasel]

Agreed, bit I dont find pragmas to be the ideal solution - its akward to use conditional directives for compilation (import/export or static library) when this is a purely linker-driven decision.
You typically already have a file for symbols and versioning, would be easiest to either directly feed that to the compiler or even better let the linker fill out the blanks correctly.

It can't be a linker thing, it has to be done at compilation time. At least if you want to avoid the thunking -- which is the only point of dllimport. If you omit it, it will still work but it will use a thunk instead of an indirect call on the address directly. (indirect call is 6 bytes, while a normal call is 5 bytes, so the linker *cannot* do anything about it without the compiler making enough "space" in the code for it)

I imagine ELF would work the same way if it had a similar thing going on, but who knows.

you cant tell a plugin to use symbols from the app (which one if multiple are used). AFAIK thats neither possible with COFF.

Multiple what? Symbols? That shouldn't be a problem at all. If you mean multiple apps (for example, a host app for a specific plugin interface), well, then such a plugin needs to use runtime loaded interface (dlopen/dlsym), since it can't statically link to a given app. That's just common sense.

Static linking means that you link specifically and depend specifically on *one* thing, hence the name "static". The dependency is baked at compile time, you can't just "hotplug" the plugin or something like that. Can't even load it into two apps at once (when you should be able to).

LTO is quite troublesome as there are have been bugs in compiler/linker implementing it aswell.

I know, I just said it's full of bugs?

Extensions like gnu symbol versioning exist eg. "perror@GLIBC_2.2.5", that could be described as namespace. whether you iterate over loaded dlls (namespaces) then iterate symbols or iterate over global symbols with namespace information should not matter.

Indeed but it's not forced and that's the problem. If symbol versioning was forced then we wouldn't have this --as-needed fiasco in the first place and nothing would break.

The "DLL Hell" is a known issue in windows, solved by pretty much any app using their preferred or even private version of DLLs, I would argue that these makes whatever issues you face pretty rare (and not because of technical superiority of COFF).

Well that's a different thing though, not loader conflicts, just DLL subtle incompatibilities. It's more about the implementation (i.e. library itself) than the container it's in.

For example with ELF, you can't even use a "private version" of a .so which is even worse.

Imagine if you bundle a private version of GTK2 with the app, only to find out that another dependency of your app depends on GTK3 and then you end up with a fiasco in the same addressing space. Meanwhile on an older distro, the app's dependency doesn't depend on GTK3 so there it ends up working just fine. This is a problem only with ELF so it's stuff like this that makes flatpak's sandboxing a necessity (amongst other reasons) which is simply not needed with DLLs/COFF.

With ELF you literally need to supply the entire runtime with the app, down to the C library (even libmesa) to be absolutely sure that stuff won't clash. That's my problem with it.

[Weasel]

Each to their own I guess, there are upsides to namespaces and there are downsides to namespaces, but that was not my point anyway. My point is that you should drop this emotional connection you have with ELF and realise that the on-disk format have exactly nothing to do with this. You can export only the exported symbols in ELF just fine and you can export every single internal symbol in PE just fine as well.

Yeah, but it's not the default, and that's not my problem with ELF, because at least it can be fixed. In that case, it's the bad libraries/projects that are at fault for relying on "exporting everything" which is just dumb.

My problem with ELF isn't the on-disk format but the fact that symbols can clash from two different libraries. Which happens fairly often if you use the same library but different "major version". Again, note how even bundling the .so file with the app is not guaranteed to work with ELF. The app could depend on other .so files which you don't bundle (because then you'll have to bundle everything they depend on, recursively, i.e. the entire runtime!) which might clash, it's out of your control, it's just an insane spaghetti mess.

So you either bundle, literally, the entire runtime, or nothing at all. That's the problem with ELF. There's no "just bundle X library and let everything else be linked from the distro's libs" middle.

[F.Ultra]

Yeah, that has other drawbacks, as now every function call is now explicitly using this struct, if you want back/forwards compatibility then things get even more complicated.
With ELF symbols + versioning you have that solved and could decide at link-time which functions will get statically linked and which are from the app.

Yeah this is one major drawback of course. That specific solutions always have both drawbacks and advantages are something that people often miss or ignore. Quite noticeable in this thread actually.

[Weasel]

Yeah, that has other drawbacks, as now every function call is now explicitly using this struct, if you want back/forwards compatibility then things get even more complicated.
With ELF symbols + versioning you have that solved and could decide at link-time which functions will get statically linked and which are from the app.

One word: dlsym.

[F.Ultra]

Yeah, but it's not the default, and that's not my problem with ELF, because at least it can be fixed. In that case, it's the bad libraries/projects that are at fault for relying on "exporting everything" which is just dumb.

My problem with ELF isn't the on-disk format but the fact that symbols can clash from two different libraries. Which happens fairly often if you use the same library but different "major version". Again, note how even bundling the .so file with the app is not guaranteed to work with ELF. The app could depend on other .so files which you don't bundle (because then you'll have to bundle everything they depend on, recursively, i.e. the entire runtime!) which might clash, it's out of your control, it's just an insane spaghetti mess.

So you either bundle, literally, the entire runtime, or nothing at all. That's the problem with ELF. There's no "just bundle X library and let everything else be linked from the distro's libs" middle.

Which again you can fix by using versioned symbols. ELF is not not the culprit behind your specific problems, the default settings in the build tools are. And once again I would state that the main problem isn't even any of those, it's the damn library writers that cannot stick to a stable ABI (there actually should be no reason a GTK2 app should not be able to use the GTK3 libraries).

However if you ever have done binary distribution you know that all of this is smoke and mirrors since you must bundle everything anyways, there is just no way for you to know if any of the libs that you are using (even if we ignore versions completely) is installed on the end users computer. This is how it's done on WIndows so even there it does not matter that PE handles your dependencies out of the box.