Fedora 29 Might Make Change To Eliminate Unnecessary Linking

[Weasel]

Now this was just a quick example that I created just for this moment. strdup() is part of the C runtime library even though Microsoft in their infinite wisdom have decided to make it "deprecated" in newer versions of VS where they have renamed it as _strdup()... https://docs.microsoft.com/en-gb/cpp...-wcsdup-mbsdup

I looked it up. It's not part of the standard library, it's POSIX only. And what library am I supposed to link with, -lmsvcrt? (I'm using MinGW, Visual C++ to me it's a very stupid compiler that breaks standards left and right and I dislike it)

How do you know you #include the same runtime when you include the standard library (free) and the POSIX runtimes (_strdup or w/e)?

Does your code even crash at all with malloc? It didn't crash here, neither on Windows VM, nor on Wine, when I used malloc. At this point I'm passing the blame on strdup being from a different runtime than wherever free gets pulled from.

Global variable? I don't know where you get any reference to global variables from. When I first found out about this problem in Windows some 15 years ago (when debug compiling a program and suddenly got a crash in something like the free() from my example above) there where MSDN articles about this, for some reason I cannot find them anymore (it's been quite a while) but atleast they still have this one: [URL="https://docs.microsoft.com/en-us/previous-versions/visualstudio/visual-studio-2012/ms235460(v=vs.110)"]https://docs.microsoft.com/en-us/previous-versions/visualstudio/visual-studio-2012/ms235460(v=vs.110) :

Which neatly describes the very thing that I see even though there once (or still do, just that I cannot currently find it) where an article on how the process thead changed runtime context. However the main problem here is that what is described in the quoted text above also can happen inside 100% your own code as in my small example.

No, it doesn't, you don't understand the article at all. (btw there is nothing about any thread or context switches, that's insane to even think about...). What I assumed about the "runtime context" was a global var switch or so. You know it's hard for me to understand what you're even talking about when you don't know it exactly yourself...

When a DLL, or .so, or whatever, calls a callback function, it's in the exact same process and address space. It is literally just a normal call instruction, just as if you did it directly from your code. You have to understand that there is more than one standard library runtime. Let's call them separately (as if with elf versioning) to illustrate the point.

Your app or library links to a given runtime, so let's say it calls malloc@abc but in code it's just "malloc". malloc@abc uses an internal structure like this:

struct malloc_abc_state
{
  size_t size;
  char data[];
};

And it returns &data[0] to you when you use malloc after doing the allocation magic (not important). free has to be implemented to use the struct above when "freeing" chunks in the heap -- it has to have the size_t field exactly there and so on. It must match exactly.

Then you have a library which you'll use its callback and it uses its own malloc/free, let's call them malloc@xyz and free@xyz.

struct malloc_xyz_state
{
  int extra_data;
  size_t size;
  int more_stuff;
  char data[];
};

Clearly they're incompatible. What sense does it make to use free@abc on a malloc done via malloc@xyz? The struct will be corrupted and it will crash. They're different functions.

But, and here's the thing: all libraries live within the same address space. When the library uses malloc under Windows, it uses malloc@xyz in this example. But all of YOUR calls to malloc are malloc@abc. You also call free@abc at the end.

Since the callback function that you gave it is YOUR code (literally part of your module), it is supposed to call malloc@abc because that's what the dynamic linker will replace it with. What you pass to the library is the address of YOUR code, your code which uses malloc@abc and not malloc@xyz.

So there should be no conflict whatsoever, unless you messed something up with strdup. Like I said did you test with malloc? I get no crash.

Yes this have exactly yo do with Windows unless we talk about objects (which I escpecially mentioned in the post you quoted so why you now try to bring up allocated objects is insidious).

Not every library creates objets, some of them manipulate i.e strings or return created void * arrays. Having a library-specific free() function for such standard C data types are beyond stupid unless you use the library on Windows where it's required for their problem of libc-version mixing.

No, having a library-specific free() function is the best way to design it because 1) it respects the language (your "unix way" violates ODR, see my previous post), and 2) it allows the library to expand its internal representation without breaking the app, transparently.

Here is a perfectly fine hash-library: https://launchpad.net/hashit where you can get the complete list of keys or values as a NULL terminated array. Since the library is created on Unix and are only used on Unix it has not idiotic x_free() function which of course means that you cannot use these two functions in Windows without a potential crash.

You can get a potential crash on Unix because it violates the C/C++ language rules. Just because it doesn't crash in YOUR case doesn't mean it's correct code.

For example, if you make an app with Clang and link a library built with GCC and do this insane thing, it will crash. That's beyond absurd, it's pathetic and stinks of trash code.

Also thanks for giving some examples, those libraries deserve to be in the "hall of shame". Your ne_free is even worse since it's a macro which is substantially different behavior than a function, so code relying on that shameful library won't even be guaranteed the same behavior.

This code:

int foobar()
{
  int ne_free; unsigned int free = 42;
  /* ... */
}

Will fail to compile on code that uses that stupid library on Unix but not on Windows, isn't that just great design?

Very "solid" library design. I can imagine they put so much thought into it /sarcasm

Shining standard of code there. Of course, Linux libs never cease to amaze me at their poor quality, so whatever.

[F.Ultra]

I looked it up. It's not part of the standard library, it's POSIX only. And what library am I supposed to link with, -lmsvcrt? (I'm using MinGW, Visual C++ to me it's a very stupid compiler that breaks standards left and right and I dislike it)

How do you know you #include the same runtime when you include the standard library (free) and the POSIX runtimes (_strdup or w/e)?

That might be correct, all I know is that strdup() is in my K&R which is how the C language once was defined. Also before Microsoft decided to deprecate things (note that I wrote that this is about 15 years ago that I noticed this?) you could use strdup() with msvcrt just fine.

And yes when I make Windows versions of the parts of our software library that we distribute to customers I also use MingGW and GCC (Visual Studio 6 was the last sane C compiler from Microsoft).

Does your code even crash at all with malloc? It didn't crash here, neither on Windows VM, nor on Wine, when I used malloc. At this point I'm passing the blame on strdup being from a different runtime than wherever free gets pulled from.

Yes it crashed on straigt malloc() aswell back in 2002/2003. Note that in order for the crash to happen your exe have to link with a different msvcrt than that of the DLL where the callback is called from.

After some more research it seams that Microsoft made changes as of VS2012 however where the heap uses the process heap instead of the private heap which probably changes things. As I said this is something that I noticed 15 years ago.

No, it doesn't, you don't understand the article at all. (btw there is nothing about any thread or context switches, that's insane to even think about...). What I assumed about the "runtime context" was a global var switch or so. You know it's hard for me to understand what you're even talking about when you don't know it exactly yourself...

I never talked about a context switch but of a switch of context, that is two completely different things. I also specifically wrote that the linked article didn't describe the switch so HTF did I not understand it? Do you not read the posts that you are commenting on?

There used to be a MSDN article about this very thing, this is not a conspiracy theory that I pulled out of my ass. I noticed that memory allocations in callback functions lead to crashed when they where later freed if the exe and the DLL where linked with different versions of msvcrt (e.f if one where linked with MSVCRT.DLL and the other linked with MSVCRTD.DLL) and after reasearching the topic then 15 years ago I stumbled upon a MSDN article from Microsoft describing exactly what I have been writing the whole time (that when a process calls a DLL then the context of which C runtime to use is switched to the one the DLL is linked with).

For some reason this article is not there anymore, perhaps they have since then changed how this works or perhaps they simply removed it when they rewrote their entire site (lot's of old MSDN links now only get you to a default site with ads for MS Surface).
[/QUOTE]

When a DLL, or .so, or whatever, calls a callback function, it's in the exact same process and address space. It is literally just a normal call instruction, just as if you did it directly from your code. You have to understand that there is more than one standard library runtime. Let's call them separately (as if with elf versioning) to illustrate the point.

Your app or library links to a given runtime, so let's say it calls malloc@abc but in code it's just "malloc". malloc@abc uses an internal structure like this:

struct malloc_abc_state
{
size_t size;
char data[];
};

And it returns &data[0] to you when you use malloc after doing the allocation magic (not important). free has to be implemented to use the struct above when "freeing" chunks in the heap -- it has to have the size_t field exactly there and so on. It must match exactly.

Then you have a library which you'll use its callback and it uses its own malloc/free, let's call them malloc@xyz and free@xyz.

struct malloc_xyz_state
{
int extra_data;
size_t size;
int more_stuff;
char data[];
};

Clearly they're incompatible. What sense does it make to use free@abc on a malloc done via malloc@xyz? The struct will be corrupted and it will crash. They're different functions.

But, and here's the thing: all libraries live within the same address space. When the library uses malloc under Windows, it uses malloc@xyz in this example. But all of YOUR calls to malloc are malloc@abc. You also call free@abc at the end.

Since the callback function that you gave it is YOUR code (literally part of your module), it is supposed to call malloc@abc because that's what the dynamic linker will replace it with. What you pass to the library is the address of YOUR code, your code which uses malloc@abc and not malloc@xyz.

So there should be no conflict whatsoever, unless you messed something up with strdup. Like I said did you test with malloc? I get no crash.

Yes that is how I assumed that it worked back 15 years ago when I stumbled upon this but both experiments and MSDN told me otherwise (at least then). In the callback malloc@xyz where called because the runtime switch only happens when the process calls the DLL and not when the DLL calls the process.

If this is still the case in 2018 I do not know since I have not tested it in years (I don't do active devlopment in Windows for some years now) but back then it worked exactly how I have now written several times.

I will however try and get by work tomorrow, I have an instance of Windows 10 inside a Virtualbox on my workstation there so I'll see if I can get this to either work or not (in case they have fixed it since). Either way I'll post back once I have done that.

[Weasel]

That might be correct, all I know is that strdup() is in my K&R which is how the C language once was defined. Also before Microsoft decided to deprecate things (note that I wrote that this is about 15 years ago that I noticed this?) you could use strdup() with msvcrt just fine.

And yes when I make Windows versions of the parts of our software library that we distribute to customers I also use MingGW and GCC (Visual Studio 6 was the last sane C compiler from Microsoft).

I was asking because I don't know what library to link to, to be able to replicate your crash. Of course, here's my theory of what happened in your case and why it crashed:

You used the standard library, which is god knows where, maybe even statically linked on Windows. So free() was statically linked or imported from wherever into your app. Let's call it free@static for illustration purposes.

But since it couldn't find strdup in the static standard library (because it's not part of the standard library), it found it in msvcrt.dll and you somehow linked to that piece. Since msvcrt.dll is a completely different implementation than free@static and has its own free@msvcrt, clearly they are incompatible as I described.

So, nice crash happened, as strdup@msvcrt is invalid for free@static.

At least that's my theory. Either way, nothing to do with DLL problem, it's user mistake (albeit a nasty one).

Yes it crashed on straigt malloc() aswell back in 2002/2003. Note that in order for the crash to happen your exe have to link with a different msvcrt than that of the DLL where the callback is called from.

After some more research it seams that Microsoft made changes as of VS2012 however where the heap uses the process heap instead of the private heap which probably changes things. As I said this is something that I noticed 15 years ago.

I honestly doubt it, probably something like I explained above, but I can't know since I can't reproduce it. I'm not doubting your crash, I'm doubting that it was this magical conflict that is not possible with my knowledge (of assembly btw, not talking about C). I'll explain below.

I never talked about a context switch but of a switch of context, that is two completely different things. I also specifically wrote that the linked article didn't describe the switch so HTF did I not understand it? Do you not read the posts that you are commenting on?

There used to be a MSDN article about this very thing, this is not a conspiracy theory that I pulled out of my ass. I noticed that memory allocations in callback functions lead to crashed when they where later freed if the exe and the DLL where linked with different versions of msvcrt (e.f if one where linked with MSVCRT.DLL and the other linked with MSVCRTD.DLL) and after reasearching the topic then 15 years ago I stumbled upon a MSDN article from Microsoft describing exactly what I have been writing the whole time (that when a process calls a DLL then the context of which C runtime to use is switched to the one the DLL is linked with).

For some reason this article is not there anymore, perhaps they have since then changed how this works or perhaps they simply removed it when they rewrote their entire site (lot's of old MSDN links now only get you to a default site with ads for MS Surface).

But there's no context, man. I'll try to illustrate what *really* happens when you use a callback.

When you call a function, in CPU instructions, you specify a destination address. A RAM address, because code lives in RAM, just like data. It's made up of bytes and so on. Then, the call pushes the return address (the code address after the call instruction) onto the stack and jumps to that address called. Upon a return instruction, it pops the return from the stack and jumps there, effectively "continuing" after the call instruction.

When you implement a callback (note: I'm talking about IMPLEMENTING a callback in code) you simply create a normal function, like this:

void my_callback()
{
  // do the strdup thing here, ofc omitted for simplicity
}

This function exists in code like any other function. It exists in YOUR code as a sequence of instructions. There's no "switch" happening, it's a simple, normal function. Heck, you can even CALL it from your own code and all it will do will be a simple CALL instruction (unless it gets inlined).

So for now I'm ignoring the library. Let's look at what the callback does:

; some assembly code here
call strdup
; more assembly code
ret

It's a simple normal function, right? For now we have no library, just this function. When your module gets loaded, this function is not treated any different. So it simply places strdup's address into the import table, just like any other function. This will be your strdup@static (in this case), which matches that of free@static. Remember, no library was loaded yet, except the standard library. So now there's no conflict because we didn't load another runtime, we agree so far.

Now Windows loads the other library, which loads the other runtime. Your function and strdup DO NOT change, they've already been fixed and imported. Your function is "loaded" already and exists the way it was when it got loaded.

When you pass the callback to the new library, you simply give its address to the library. In this case you give the address of "my_callback" (32-bit or 64-bit data) to the library's parameter that takes it (a pointer). When you call this library and pass it the callback, all you do is:

call library_func

where library_func is imported by the loader. There's no switch yet.

Later, when the new library (using the new runtime) decides to call your callback, all it does is:

call *ptr

where "ptr" holds your "my_callback" address.

That's... it.

Literally.

There's no switch. The strdup@static shouldn't even be aware it got called from a callback: all it knows (via return address) is that it was called from my_callback, which is a normal function. In fact, you can even call my_callback from YOUR own code, and the behavior is completely identical. Even the return address will be identical (it's my_callback's) and strdup won't know the difference.

This is why to me it is literally impossible to conflict here. Conflicts happen during load time -- when the loader (DLL or ELF) replaces symbols with their actual addresses. The problem with ELF is that a symbol may be used by more than 1 library, that's literally its only problem. In this case it will conflict during LOAD (but the crash will happen later when it's called of course).

ELF added versioned symbols (that very few libraries use) precisely to combat this problem. It makes them more like DLLs, but it's a bit worse still.

With ELF, it's like placing all symbols as global variables, while with DLL, it's like placing them each in their own separate class or namespace, given by the module. e.g.

// ELF
symbolA;
symbolB;
symbolC;

symbolA;  <-- conflict


// DLL
namespace module1
{
  symbolA
  symbolB
}
namespace module2
{
  symbolC
}
namespace module3
{
  symbolA
  symbolC
}

You know, it doesn't really take so long to realize which one is vastly superior here.

The former is a complete total spaghetti mess and extremely fragile the more libraries you pull in. Now try it with loading the same library but different versions: DLL has no conflict (separate namespaces) while with ELF it's guaranteed to conflict.

[timofonic]

Actually, looks like Arch Linux makepkg.conf includes it by default.

So Arch Linux and Gentoo are pioneers on this?

[coder]

Go post your answer on stackoverflow and watch it down votted to the abyss.

You're mistaken if you think it counts for anything that you can find other Windows programmers with the same opinions.

Who the fuck do you think you are? Is this your site or something?

As I have 3 years and 3x as many posts, how about you indulge me and explain your purpose on this site. You've already admitted you don't use (or, at least develop on) Linux. Why should we believe you're here for any purpose other than trolling?

[coder]

Destructors are just normal functions, and if you expose them (via header) then they are automatically exposes just like a C function. Why does it matter if it's implicit or explicit? The binary is the same.

This is the second time I've tried to explain this point and the second time you've missed it. You're just steamrolling over any points that don't line up with your argument, so I give up.

This is not a Windows limitation this is a C/C++ language thing.

http://lmgtfy.com/?q=C%2B%2B+One+Definition+Rule

Your use of a data type external to your header, such as the standard library, that may not be identical, is a direct violation of ODR.

Yes, I get all that, but you don't seem to understand that it doesn't matter when your system consists entirely of opensource packages that can be rebuilt when an incompatible change is introduced in a shared dependency. Hence my point about superior version management on Linux systems.

You're so stuck in your closed-source Windows mindset that you can't (or won't) see any other way.

[Weasel]

You're mistaken if you think it counts for anything that you can find other Windows programmers with the same opinions.

Nice strawman you got there. stackoverflow is not a Windows site nor about Windows programming in general. Don't be depressed that the C/C++ language itself forbids your "Linux way" of doing... stuff... it's not Windows' fault that ELF fucked up the userland.

As I have 3 years and 3x as many posts

https://www.logicallyfallacious.com/...alse-Authority

We should take misinformation and posts not backed by FACTS (e.g. links) out of the equation and instead count as a negative, that would paint a clearer picture. Spamming claims isn't exactly educational for readers.

how about you indulge me and explain your purpose on this site.

Spreading facts and calling bullshit for what it is, so that viewers are not drawn by fanboys' logic and false claims. I mean it's public, so obviously it's for viewers to read.

You've already admitted you don't use (or, at least develop on) Linux. Why should we believe you're here for any purpose other than trolling?

This is the kind of nonsense I'm talking about. I said I don't use ELF unless I have to, and most certainly do not develop for ELF because it's archaic and retarded. Linux can use DLLs, it's called Wine, which I've already stated before, but not like you read anything.

Also this is NOT a Linux forum, it's an open source forum (about open source I mean). Wine is open source project that can load DLLs. Got a problem with that? Sounds perfectly valid to me.

I don't use Windows (except in VMs when I have to). I use Linux with a sane environment called Wine, because Linux userland is just garbage and insane, and only good when it's all compiled together (i.e. base packages of a distro, not for keeping your apps around). This entire thread proves it, the fact Fedora has to even consider doing such a thing in the first place just STINKS of piss poor userland and design.


EDIT: Another post awaiting moderation because I posted two links in it... blah. Michael please consider changing the spam filter a bit, people need links to back their points <.<

[Weasel]

This is the second time I've tried to explain this point and the second time you've missed it. You're just steamrolling over any points that don't line up with your argument, so I give up.

Show me disassembled C++ destructor code and then a "normal" function call and let's look at the difference (ofc destructor would have a mangled name based on its class and an implicit parameter for the object). If I show you any, you'll say I compiled the wrong code, so go show it champ.

You can use gcc.godbolt.org to provide proof too.

I mean DLLs and ELF don't give a shit about the code, only the binary result, that's what they hold, so...

Yes, I get all that, but you don't seem to understand that it doesn't matter when your system consists entirely of opensource packages that can be rebuilt when an incompatible change is introduced in a shared dependency. Hence my point about superior version management on Linux systems.

You're so stuck in your closed-source Windows mindset that you can't (or won't) see any other way.

WTF kind of argument is this? Just because it might not be an issue when you can "recompile everything" it doesn't mean it's BETTER it just means you IGNORE THE PROBLEM. DLLs have NO issue if you "recompile everything" either so how is this an argument for ELF? I'm not looking for where "ELF works fine" but rather where ELF is superior to DLLs and the fact is that it never is.

Actually as you can see, it's "not fine" even when you "recompile everything" because Fedora has to add this change in the first place. It slows down performance and loads extra libraries currently, so no, it's "not fine" at all. A problem that wouldn't exist with DLLs has to be fixed now in this thread and Fedora because ELF is just bad and has a problem that DLL doesn't.


Another retarded Unix/Linux thing is default public visibility. Microsoft got it right with default private visibility, and I have to admit facts when I see them.

And no, don't start with your fanboyism again. Even the GCC developers agree with this because they're sane, see: https://gcc.gnu.org/wiki/Visibility. In short, public visibility not only leads to conflicts, it results in worse code overall, so yes in fact much Windows code *is* faster in many cases than on Unix due to shit decisions done ages ago.

Not everything Unix did was right, just like how not everything Microsoft did was right. For example the registry is the most vile pile of garbage ever invented (at least for applications' use, for system use it's fine). We would've solved these issues long ago if fanboys didn't cry all the time about changing their beloved (but flawed) way. Also Microsoft's lack of symbolic links until Vista was another disaster where Unix is just superior, not to mention drive letters being crap. Yeah I'm a super Windows fanboy indeed. /sarcasm

[moonshadow565]

Depends on which context you are thinking about regarding differences. If we are talking about the very topic that is discussed in this thread then the executable format does not make any difference (i.e unnecessary linking is done at the linking stage and not due to the format of the file). The major difference between Mach-O and ELF is that Mach-O can store multiple binaries for different architectures in the same file (which ELF can do with an extension).

I'm aware of that it can load multiple architectures which is sadly only extension on elf (which afaik isn't accept into mainline linux kernel).
I am more interested in multiple libraries per process problem that has beeb discused on this thread...
Acording to wikipedia Macho-o has something called 2 level namespace, is this related to this probelm?
The darling project implements Macho-o loader for linux so i would be more interested to see that as alternative to elf than dll.
Hope some experts can shed some light on this

[F.Ultra]

I'm aware of that it can load multiple architectures which is sadly only extension on elf (which afaik isn't accept into mainline linux kernel).
I am more interested in multiple libraries per process problem that has beeb discused on this thread...
Acording to wikipedia Macho-o has something called 2 level namespace, is this related to this probelm?
The darling project implements Macho-o loader for linux so i would be more interested to see that as alternative to elf than dll.
Hope some experts can shed some light on this

The two-level namespace in Mach-o (the name is Mach-o and not Macho-o) means that each imported symbol have their corresponding library defined.

Lets say that we have one process A and three shared libraries B, C and D.

Process A links with B to use function1() and with C to use function2().
Library C links with D to use function1() in it's function2().

We now have two different functions named function1() (one from B and one from D).

In a non-versioned ELF situation we end up with function1() from B being used both when it's called from A and when it's called from function2() in C.

On Mach-o however since each exported symbol have their respective library name attached function1() from B will be used when called from A and function1() from D will be used when called from function2() in C.