Fedora 30 To Take Stab At Eliminating Excessive Linking

[F.Ultra]

One word: dlsym.

And you can use that on .exe files on Windows?

[discordian]

One word: dlsym.

Thats again explicit in code, and AFAIK wont inter-operate with symbol versioning. ELF relocation can further be listed by tools and are ready-to-run without any setup or other explicit state or trickery.
As concrete example: I used ELF DSOs on embedded systems without libdl, and depending on CPU some functions like sqrtf where either linked statically in the plugin (small code with FPU), or did relocate to the applications copy of sqrtf (lotsa code because of no FPU). The plugins code does not know where the function comes from, the only difference is whether its statically linked or not (the app then export this function). Bonus point: the compiler can optimize better and potentially even remove calls to sqrtf as this is a known function.

Sorry that's more than one word, but there aint a clear solution for everything.

[discordian]

It can't be a linker thing, it has to be done at compilation time. At least if you want to avoid the thunking -- which is the only point of dllimport. If you omit it, it will still work but it will use a thunk instead of an indirect call on the address directly. (indirect call is 6 bytes, while a normal call is 5 bytes, so the linker *cannot* do anything about it without the compiler making enough "space" in the code for it)

It can, if the linking stage can rewrite code, its a form of pretty light LTO. Or you could take LLVMs approach which uses (close-to-assembly) byte-code instead of assembly,
function calls arent resolved to lowest level instructions there.

I imagine ELF would work the same way if it had a similar thing going on, but who knows.

Err. what?

Multiple what? Symbols? That shouldn't be a problem at all. If you mean multiple apps (for example, a host app for a specific plugin interface), well, then such a plugin needs to use runtime loaded interface (dlopen/dlsym), since it can't statically link to a given app. That's just common sense.

multiple apps, or different names - point is the linker wont ever resolve to it anyway.
Statically linking is not relevant, dynamic linking is missing a correct name. Ideally would want to specify an interface that the App implements.
Short of that option, checking the dangling symbols to be resolved by the app is enough.

Static linking means that you link specifically and depend specifically on *one* thing, hence the name "static". The dependency is baked at compile time, you can't just "hotplug" the plugin or something like that. Can't even load it into two apps at once (when you should be able to).

Typically you have a static and/or dynamic library, you can choose to build your code against either, but as you correctly realized not at the same time.

I know, I just said it's full of bugs?

I thought you meant the apps have buggy code that only triggers with LTO, not that LTO has bugs.

Indeed but it's not forced and that's the problem. If symbol versioning was forced then we wouldn't have this --as-needed fiasco in the first place and nothing would break.

Coding guidelines and unit tests aren't forced either, little excuse not to use it.

Well that's a different thing though, not loader conflicts, just DLL subtle incompatibilities. It's more about the implementation (i.e. library itself) than the container it's in.

I guess if someone builds an ELF file it worked on his system and broke down somewhere else. The point is that on Windows any shortcomings of Coff are unlikely to show if the App ships with the complete system (exaggerated)

For example with ELF, you can't even use a "private version" of a .so which is even worse.

err, I do this regularly. rpath or LD_LIBRARY_PATH.

Imagine if you bundle a private version of GTK2 with the app, only to find out that another dependency of your app depends on GTK3 and then you end up with a fiasco in the same addressing space. Meanwhile on an older distro, the app's dependency doesn't depend on GTK3 so there it ends up working just fine. This is a problem only with ELF so it's stuff like this that makes flatpak's sandboxing a necessity (amongst other reasons) which is simply not needed with DLLs/COFF.

Well, thats exactly DLL Hell, and Windows has Side-by-side assembly for that reason.

With ELF you literally need to supply the entire runtime with the app, down to the C library (even libmesa) to be absolutely sure that stuff won't clash. That's my problem with it.

So if symbols don't clash, then you can still have different behavior between DLLs breaking your App, conversely if behavior is identical then it does not matter where the symbol is coming from.

Its missing versioning (be it symbol or soname) or broken forwards-compatibility, plain and simple

[Weasel]

And you can use that on .exe files on Windows?

GetProcAddress? Unless you mean the same code, then just use a simple wrapper like dlfcn-win32

[Weasel]

Which again you can fix by using versioned symbols. ELF is not not the culprit behind your specific problems, the default settings in the build tools are. And once again I would state that the main problem isn't even any of those, it's the damn library writers that cannot stick to a stable ABI (there actually should be no reason a GTK2 app should not be able to use the GTK3 libraries).

Dude how many times must I say it that the format should *enforce* this thing. It's not a question of defaults. It's a question of making it *impossible* to do it otherwise. Why is that important? Because poorly written crap will remain poorly written crap. Global variables are the "easiest" way to write code, and so are symbols in the global namespace and "Exporting everything". You don't have to think about design and a proper interface, and then you end up with fiasco later (for global vars, probably threaded issues, while for ELF it's the symbol clashes etc).

I mean the format should, literally, require a library name to be attached to a list of symbols. Just like dlopen and dlsym.

However if you ever have done binary distribution you know that all of this is smoke and mirrors since you must bundle everything anyways, there is just no way for you to know if any of the libs that you are using (even if we ignore versions completely) is installed on the end users computer. This is how it's done on WIndows so even there it does not matter that PE handles your dependencies out of the box.

This couldn't be more wrong. At least half of all the libraries in most apps will be system DLLs, nobody ships those. Nobody actually ships Windows DLLs except the distributable runtimes (Visual C++ runtimes or DirectX), or when they need them "patched" for certain hacks (most famous is ddraw.dll for old games). You won't see stuff like gdi32.dll or user32.dll or gdiplus or whatever shipped.

If someone does ship gdiplus it's probably for extremely old version of Windows: but they don't ship the entire system32 directory.

[Weasel]

It can, if the linking stage can rewrite code, its a form of pretty light LTO. Or you could take LLVMs approach which uses (close-to-assembly) byte-code instead of assembly,
function calls arent resolved to lowest level instructions there.

Sorry but LTO is compilation so it doesn't count here. Linking can't modify code unless it fits in the same amount of bytes (and then it patches some offsets).

Err. what?

When you import from a DLL with dllimport, the compiler generates code like:

call [ExitProcess]

which is a 6-byte call that reads from the IAT and calls the pointer there. Without dllimport, the compiler will generate

call ExitProcess

because it thinks it's a normal function about to be linked via an object file or a static library. This instruction is 5 bytes.

So when the linker sees it's an external symbol, it adds a thunk (a jump) and points the call there. This jump gets patched later to jump to the actual symbol at runtime, etc.

ELF, as far as I know, uses only the latter approach. So I simply said, if it had something akin to "dllimport" then it would probably use a similar mechanism.

multiple apps, or different names - point is the linker wont ever resolve to it anyway.

Then use dlopen and dlsym, that's what they are for. Linking done at "build time" is a static process. All the dependencies must be baked precisely at the moment of compilation/linking, not at runtime.

If you need the plugin to be hotplugged into any compatible app implementing the interface (e.g. an audio plugin), then you simply cannot bake the dependency at build time, it must be determined at runtime, which is why dlopen and dlsym exist, among other reasons! It's same with "optional" libraries: some dependencies are simply done at runtime, and plugins like this are one of them.

Anything else is a stupid hack and should not be allowed to work, because it's counter to logic. People just want it because "it's easier" that's why so many projects don't even have a proper design because it's "easier" without a design, while the code quality sinks down.

I guess if someone builds an ELF file it worked on his system and broke down somewhere else. The point is that on Windows any shortcomings of Coff are unlikely to show if the App ships with the complete system (exaggerated)

Where do you guys get this stupid idea that Windows applications ship with the complete system (or close to it?). Did you even look how many system DLLs it actually links against? The point is that there's two things that matter here: the library implementation API/ABI, and the underlying loader problem. We're talking about the latter.

The only apps which bring a LOT of 3rd party libs are those using "posix" libraries or other "cross platform" stuff, but that's due to their choice. POSIX is not God, Microsoft have their own libraries for most of that stuff. Just because Linux uses those libraries doesn't make them "true" libraries everything should be judged against.

Think about this: If applications shipped with 90% of the Windows system bundled with them, then the Wine project would have been finished eons ago since the "system" itself would require little code. It's just patently false, stop spreading this misinformation.

err, I do this regularly. rpath or LD_LIBRARY_PATH.

Yeah, no. I already said why it's ineffective and it can clash, due to ELF's design. LD_LIBRARY_PATH would work just fine with DLLs though.

Well, thats exactly DLL Hell, and Windows has Side-by-side assembly for that reason.

No, DLL Hell is about library incompatibilies, not symbol clashes. It's a library design problem, not a loader/format problem.

With ELF you can have BOTH problems. ELF doesn't forbid people from writing unstable libraries, oh wait, Linux is FULL of them! But ELF also suffers from symbol clashes, even for stable libraries.

And my problem with ELF is clearly not the "unstability" of the libraries, because that's the libraries' fault and another of my complaints but not this topic. My problem is the latter, which is exclusive to it.

[F.Ultra]

Dude how many times must I say it that the format should *enforce* this thing. It's not a question of defaults. It's a question of making it *impossible* to do it otherwise. Why is that important? Because poorly written crap will remain poorly written crap. Global variables are the "easiest" way to write code, and so are symbols in the global namespace and "Exporting everything". You don't have to think about design and a proper interface, and then you end up with fiasco later (for global vars, probably threaded issues, while for ELF it's the symbol clashes etc).

I mean the format should, literally, require a library name to be attached to a list of symbols. Just like dlopen and dlsym.

With that logic PE is completely broken since you can link by ordinal instead of by name if you like to which IMHO create far greater problems than non-versioned symbols.

This couldn't be more wrong. At least half of all the libraries in most apps will be system DLLs, nobody ships those. Nobody actually ships Windows DLLs except the distributable runtimes (Visual C++ runtimes or DirectX), or when they need them "patched" for certain hacks (most famous is ddraw.dll for old games). You won't see stuff like gdi32.dll or user32.dll or gdiplus or whatever shipped.

If someone does ship gdiplus it's probably for extremely old version of Windows: but they don't ship the entire system32 directory.

And why do you assume that we are talking about the system libraries? No one have suggested to bundle any system libraries.

GTK2 and GTK3 are not system libraries. glibc could be considered a system library and there you can run a 20 year old application on a glibc from 2018 without any problem so bundling is not necessary. You might argue that a problem with Linux is that we have no system libraries for GUI such as Gtk or Qt and that Linux is heavily fragmented in this area when Windows provides a single interface but that have nothing to do with versioned symbols or ELF.

The reason you don't see so many bundled libraries for games and apps on Windows is because they often link statically which is why the main .exe is usually so huge.

[Weasel]

With that logic PE is completely broken since you can link by ordinal instead of by name if you like to which IMHO create far greater problems than non-versioned symbols.

Then you have to be careful and keep the ordinals the same (with a .def file, not very difficult).

And why do you assume that we are talking about the system libraries? No one have suggested to bundle any system libraries.

GTK2 and GTK3 are not system libraries. glibc could be considered a system library and there you can run a 20 year old application on a glibc from 2018 without any problem so bundling is not necessary. You might argue that a problem with Linux is that we have no system libraries for GUI such as Gtk or Qt and that Linux is heavily fragmented in this area when Windows provides a single interface but that have nothing to do with versioned symbols or ELF.

Sigh, I'm afraid you won't get it... but here's my last attempt at explaining to you the problem here and why ELF is just... bad.... and why DLL doesn't suffer from it (irrespective of library design). I should probably quote myself but whatever.

First, I acknowledge that glibc is stable, however that's a completely different point than what I'm making. I'm not talking about the stability of libraries themselves -- that's a problem of the library developers, ELF is not at fault here. It is one of the problems I have with Linux userland, however this is NOT the topic at hand, and I did not complain about it in this thread. Ok, I just wanted to get that out of the way so you can understand I'm not talking about "ELF should enforce stability" which makes no sense (DLL doesn't, either).

So let's assume that both the ELF library and DLL is unstable in this example. I will use GTK as the example here, but it's just a placeholder -- let's assume that GTK is a system library on Windows. Yes, with all its incompatibilities between version 2 and 3, let's assume that is so, for now. So that the first point with the incompatibility is nullified.

You have your application, "TheSuperAwesomeGame". This game has, naturally, a lot of dependencies. One of the dependencies is gtk2, another is "libdependency" which is a system library. Now, you compile this game, and it runs... on your distro/machine. Because it satisfies all dependencies. But you obviously want to run it on other distros too.

There's a problem: while all distros you care about (same as what Windows versions you care about) bundle libdependency, it being a system library and all that, and it is compatible with your game, they don't bundle gtk2.

You being a smart guy think to yourself: why don't I just bundle gtk2 with my app? We don't want to static link because we want to show why ELF is just bad. Static linking clearly "avoids" the problem, but that doesn't make ELF any good, in fact it proves it's bad that you have to resort to "avoid" it in the first place.

On Windows just place gtk2.dll in the app's directory, on Linux use LD_LIBRARY_PATH. Looks good, now TheSuperAwesomeGame runs on the new distro! It works! Cool stuff!

So ELF is not so bad? It works just like Windows and DLLs? Not yet, hold your horses, let me outline the real problem with ELF right now in the next example.

Now you want to try TheSuperAwesomeGame on another distro. This time, the same thing happens: the distro doesn't ship gtk2, and it has a compatible libdependency. Cool stuff, do the same LD_LIBRARY_PATH and... the game doesn't run. WTF you say? Maybe it's some other dependency that's missing and that's the problem? Nope, it's not (let's say there's not), because on Windows (which, in this example, uses the same libraries, but dll instead) it just works. All dependencies are satisfied and yet the game fails to run, WTF?!?

Welcome to ELF and global namespaces!

In DLL-land such a thing is unthinkable, else WinSxS would never work. In other words, DLL Hell may be a thing (a completely different topic), but the side-by-side assemblies that exist on Windows to counter it cannot even exist on ELF. ELF is just that worse: it suffers from the same DLL Hell but it cannot solve it by design. Ever.

So ok, I've been rambling now, but I didn't yet tell you what the issue with ELF is right now? Glad you're still with me. Here's the shocking revelation:

libdependency on the new distro depends on gtk3.

Yes. This particular libdependency is fully compatible in terms of API/ABI with your game, however internally it depends on gtk3. Since your game relies on gtk2 and loads it first, when it loads libdependency on this distro, it will load gtk3 as well, resulting in a fucking mess. This happens because of the global namespace: if each gtk had its own namespace, nothing would break. Therefore, TheSuperAwesomeGame works just fine on a system with DLLs, no matter what dependencies libdependency uses. But it won't on ELF, you have to *pray* that its other dependencies don't conflict with any of your bundled libs.

And here's why ELF sucks. You have to supply your own libdependency with the game itself, just to make sure it's a version that doesn't link to gtk3.

You did not have to do that on a system with DLLs -- all you had to do is supply the missing dependency, which was gtk2.dll, and that's it! You don't care what libdependency.dll links against, you simply don't! If it is compatible with your game, it will work, full stop. It's sane!

That's why I kept saying ELF is a spaghetti mess: unrelated components can conflict with others, which is the true definition of spaghetti code.

Supplying gtk2.so is not enough as you can see. It might work on some distros, but not on others, if stars align badly, it's just extremely fragile. DLL is fine, ELF is not.

Do you know what this leads to? You end up having to supply the entire runtime down to glibc, JUST TO MAKE SURE they don't load gtk3. Even though glibc is stable and compatible, it could rely on a dependency like gtk3 (could, not that it does) which fucks everything up. ELF is not just about API/ABI stability, it suffers from way more issues than DLLs.

BTW, this is exactly why flatpak has to do this. So it's a real problem that had a TON of man hours spent on it. And it still sucks. Blah.

The cause of it? Global namespace.

The reason you don't see so many bundled libraries for games and apps on Windows is because they often link statically which is why the main .exe is usually so huge.

You misunderstood the point. I was talking about the app, even statically linked, linking to a ton of Windows DLLs in the end. You can't have that on bLinux, see above why, hopefully for the LAST time as I've given this example at least a million times by now.

[F.Ultra]

Then you have to be careful and keep the ordinals the same (with a .def file, not very difficult).

Ok so when it's ELF it's insane and stupid but if it's PE then it's just "not very difficult" . See why I say that you are emotionally invested here and not applying logic?

Call me back when you discover that one or several of your clients decided (or rather their tool chains decided) to import your DLL by ordinal instead of by name and the havoc that ensures when you added a new function to said DLL without making sure that the ordinals where the same (since you never used a .def in the first place).

Sigh, I'm afraid you won't get it... but here's my last attempt at explaining to you the problem here and why ELF is just... bad.... and why DLL doesn't suffer from it (irrespective of library design). I should probably quote myself but whatever.

First, I acknowledge that glibc is stable, however that's a completely different point than what I'm making. I'm not talking about the stability of libraries themselves -- that's a problem of the library developers, ELF is not at fault here. It is one of the problems I have with Linux userland, however this is NOT the topic at hand, and I did not complain about it in this thread. Ok, I just wanted to get that out of the way so you can understand I'm not talking about "ELF should enforce stability" which makes no sense (DLL doesn't, either).

So let's assume that both the ELF library and DLL is unstable in this example. I will use GTK as the example here, but it's just a placeholder -- let's assume that GTK is a system library on Windows. Yes, with all its incompatibilities between version 2 and 3, let's assume that is so, for now. So that the first point with the incompatibility is nullified.

You have your application, "TheSuperAwesomeGame". This game has, naturally, a lot of dependencies. One of the dependencies is gtk2, another is "libdependency" which is a system library. Now, you compile this game, and it runs... on your distro/machine. Because it satisfies all dependencies. But you obviously want to run it on other distros too.

There's a problem: while all distros you care about (same as what Windows versions you care about) bundle libdependency, it being a system library and all that, and it is compatible with your game, they don't bundle gtk2.

You being a smart guy think to yourself: why don't I just bundle gtk2 with my app? We don't want to static link because we want to show why ELF is just bad. Static linking clearly "avoids" the problem, but that doesn't make ELF any good, in fact it proves it's bad that you have to resort to "avoid" it in the first place.

On Windows just place gtk2.dll in the app's directory, on Linux use LD_LIBRARY_PATH. Looks good, now TheSuperAwesomeGame runs on the new distro! It works! Cool stuff!

So ELF is not so bad? It works just like Windows and DLLs? Not yet, hold your horses, let me outline the real problem with ELF right now in the next example.

Now you want to try TheSuperAwesomeGame on another distro. This time, the same thing happens: the distro doesn't ship gtk2, and it has a compatible libdependency. Cool stuff, do the same LD_LIBRARY_PATH and... the game doesn't run. WTF you say? Maybe it's some other dependency that's missing and that's the problem? Nope, it's not (let's say there's not), because on Windows (which, in this example, uses the same libraries, but dll instead) it just works. All dependencies are satisfied and yet the game fails to run, WTF?!?

Welcome to ELF and global namespaces!

In DLL-land such a thing is unthinkable, else WinSxS would never work. In other words, DLL Hell may be a thing (a completely different topic), but the side-by-side assemblies that exist on Windows to counter it cannot even exist on ELF. ELF is just that worse: it suffers from the same DLL Hell but it cannot solve it by design. Ever.

So ok, I've been rambling now, but I didn't yet tell you what the issue with ELF is right now? Glad you're still with me. Here's the shocking revelation:

libdependency on the new distro depends on gtk3.

Yes. This particular libdependency is fully compatible in terms of API/ABI with your game, however internally it depends on gtk3. Since your game relies on gtk2 and loads it first, when it loads libdependency on this distro, it will load gtk3 as well, resulting in a fucking mess. This happens because of the global namespace: if each gtk had its own namespace, nothing would break. Therefore, TheSuperAwesomeGame works just fine on a system with DLLs, no matter what dependencies libdependency uses. But it won't on ELF, you have to *pray* that its other dependencies don't conflict with any of your bundled libs.

And here's why ELF sucks. You have to supply your own libdependency with the game itself, just to make sure it's a version that doesn't link to gtk3.

You did not have to do that on a system with DLLs -- all you had to do is supply the missing dependency, which was gtk2.dll, and that's it! You don't care what libdependency.dll links against, you simply don't! If it is compatible with your game, it will work, full stop. It's sane!

That's why I kept saying ELF is a spaghetti mess: unrelated components can conflict with others, which is the true definition of spaghetti code.

Supplying gtk2.so is not enough as you can see. It might work on some distros, but not on others, if stars align badly, it's just extremely fragile. DLL is fine, ELF is not.

Do you know what this leads to? You end up having to supply the entire runtime down to glibc, JUST TO MAKE SURE they don't load gtk3. Even though glibc is stable and compatible, it could rely on a dependency like gtk3 (could, not that it does) which fucks everything up. ELF is not just about API/ABI stability, it suffers from way more issues than DLLs.

BTW, this is exactly why flatpak has to do this. So it's a real problem that had a TON of man hours spent on it. And it still sucks. Blah.

The cause of it? Global namespace.

You misunderstood the point. I was talking about the app, even statically linked, linking to a ton of Windows DLLs in the end. You can't have that on bLinux, see above why, hopefully for the LAST time as I've given this example at least a million times by now.

Yes I got it the first time that you wrote this but it still does not mean that I don't agree with you (people can not agree while still understanding your point see). Also I have been doing binary distribution on several systems since Windows was 16-bit so I do have some experience in the matter.

You think that your problem is ELF while it in reality is that Linux does not have a common set of system libraries like Windows does. It's as simple as that. The problem is not that libdependant suddenly on some distro links to GTK3, the problem is that you cannot assume that libdependent even exists on that distro to begin with so you have no other option that to bundle or do static linking.

You have a single library that you know will exist on most distributions and that is glibc but even that is uncertain since there are distributions like Alpine where musl is used instead (however Alpine will never be the target of your app or your game since it's only existence is to be a base for flatpacks). And glibc (and musl for that matter) is self contained so it will never link to any of your dependencies.

It's this lack of a common set of system libraries that are your problem, for any other library that you use if you do binary distribution you have to either provide a distro specific build (this is what I do for every single of our non fire-and-forget binaries that we distribute) or you have to bundle everything except glibc.

This is not a problem with ELF (and if it where it could be easily be fixed with versioned symbols so yet again no problem with ELF) but that you cannot assumed the existence of a shared library on your potential host system (unless you build for a specific system like say Steam).

It just so happens that you have something like libgtk on Windows that you find this troublesome on Linux, but again this is not due to ELF but due to libgtk not being something that you can assume exists on a specific distro. And this is a major shame but also the price we pay for the freedom that Linux offers (as always there are cons and pros for every decision).

And I say this again, the major main big bada boom problem is that library writers don't #1 use versioned symbols and #2 that they have some sexual or political preference against maintaining a stable ABI. Because with versioned symbols you actually get a much more powerful tool that PE will give you since you can provide the entire "old" gtk2 interface in your new gtk3 library and support both old and new applications (both applications can now use say create_window() and one will get v2 and the other will get v3 completely transparently).

[F.Ultra]

...

Oh and btw I do not fully trust fire-and-forget distribution on Windows either. Among other things, we stumbled upon one version of zlib that if used crashed any application hard on Windows 2003 Server 32-bit but worked fine on every other version of Windows (including the 64-bit version of Windows 2003 [and that included the 32-bit DLL]) both older and newer. Never found out what that was about (the previous version worked and since the number of 32-bit customers was very low we simply cheated by bundling the older 32-bit version and the newer 64-bit version).