Fedora 29 Might Make Change To Eliminate Unnecessary Linking

[Weasel]

I think you've got the causality wrong. You're accepting Windows' constraint as given and then looking to justify it.

I've been down the road of custom allocators, a few times, and never found them justified. Especially for API-public types. But that's not even what we're talking about. We're talking about the exact opposite - a separate allocator for each library, by default, even though 99% of the time it's just the same underlying implementation. Were I to match your hyperbole, I could use some choice words to describe that architecture.

No, I'm saying that the allocator is local to a library and should be local, otherwise your library is designed poorly. Whoever does it should get fired.

An allocator is not just malloc. It includes free.

Think of C++: an allocator is a class. The class implements both new and delete operators. You don't fucking let the caller of the library to implement and call delete on YOUR class, WTF?? That's such a straw man argument from people who have nothing against Windows' superior DLL design.

Same thing in C. malloc/free are a PAIR and should always be used as SUCH. A malloc from library X must be considered distinct from library Y, and thus its free should also be distinct. Also from app Z.


Other post awaiting moderation because I destroyed some rookies with too many links I guess...

[moonshadow565]

Wow, so many smart people here!
Can someone explain to me how Macho-o compares to elf and PE?

[coder]

No, I'm saying that the allocator is local to a library and should be local, otherwise your library is designed poorly. Whoever does it should get fired.

An allocator is not just malloc. It includes free.

That's pretty much the issue. Lots of library functions allocate memory the caller is responsible for freeing, and many don't bother to wrap free(). You're using the argument of an extreme corner case where it might be justifiable to use a custom allocator to say that all memory allocation must always be wrapped, which has a real cost in runtime and API weight and forcing layers of wrapping and abstraction or copying on simple API types like strings or raw data buffers that might be passed across several library boundaries.

If you weight down your code with abstractions to handle every corner case that might apply to any library - not just yours - you run a greater risk of introducing or hiding bugs. And libraries that are cumbersome to use generally aren't.

Think of C++: an allocator is a class. The class implements both new and delete operators. You don't fucking let the caller of the library to implement and call delete on YOUR class, WTF?? That's such a straw man argument from people who have nothing against Windows' superior DLL design.

In my experience, custom new & delete operators are mercifully quite rare.

[Weasel]

That's pretty much the issue. Lots of library functions allocate memory the caller is responsible for freeing, and many don't bother to wrap free().

Such as? (give me examples of Windows libraries btw, since you used this example against Windows -- I'm not surprised if Linux libs do have this "problem" since, well, the quality of most libraries is kind of subpar)

Of course, in Windows it's perfectly fine for a library to ask you to use one of VirtualFree, LocalFree or GlobalFree, because those are system calls and always the same no matter the runtime or dependencies (i.e. things won't break). So those don't count.

You're using the argument of an extreme corner case where it might be justifiable to use a custom allocator to say that all memory allocation must always be wrapped, which has a real cost in runtime and API weight and forcing layers of wrapping and abstraction or copying on simple API types like strings or raw data buffers that might be passed across several library boundaries.

If you weight down your code with abstractions to handle every corner case that might apply to any library - not just yours - you run a greater risk of introducing or hiding bugs. And libraries that are cumbersome to use generally aren't.

Seriously? Every library worth its salt has its own free or destroy or destructor, it's just common sense practice. This is not a corner case, this is the norm. Most of the time you don't even just "wrap free", you have to do more work to begin with when destroying the library's object.

For example, zlib, flac, etc.

If you think destroying an object via an extra function call is some sort of "serious overhead" then you shouldn't be using a dynamic library in the first place, static link it instead with LTO. That's like 0.01% of the performance hit.

[Weasel]

Wow, so many smart people here!
Can someone explain to me how Macho-o compares to elf and PE?

Given the nonsense I had to reply to, I think it's safe to say I'm the only programmer in this discussion (no, you don't have to take my word for it, I supplied links to prove I know what I'm talking about if you are *really* interested).

And unfortunately for you, I couldn't care less about the Mac, so I have no idea about anything regarding it. Unless someone else chimes in, that is. Something tells me it's not any better than elf tho.

[F.Ultra]

Dude, what?

There is literally zero context switches between libraries because they're in the same address space (unless you mean different threads but that's a completely different thing). That's the whole point of libraries compared to processes in the first place. I have seriously NO idea where you got that from but it's so wrong it's not even funny.

Sorry to burst that bubble dude but this is really happening, I've programmed Windows applications for decades and have seen many young devs fall for this.

If your application is linked with runtime A and you then use a DLL that is linked with runtime B and your process calls a function in that DLL then Windows will switch the current runtime context for your thread to B and when the function returns it will switch back to A again. However the major bug here is that if code in the DLL calls back to your application then the runtime context is still B and not A.

You can test this extremely easy yourself with libraries such as cURL or Expat. E.g with Expat you define callbacks to handle when an XML-tag starts and ends:

struct XMLUSERDATA {
    char *test;
};

static void xml_end (void *udata, const XML_Char *name)
{
}

static void xml_start (void *udata, const XML_Char *name, const XML_Char **atts)
{
    struct XMLUSERDATA *userdata = udata;

    userdata->test = strdup (name); /* we allocate a string in our own code here */
}

int main (int argc, char **argv)
{
    struct XMLUSERDATA userdata;
    memset (&userdata, 0, sizeof userdata);

    XML_Parser xml = XML_ParserCreate (NULL);

    XML_SetUserData (xml, &userdata);
    XML_SetElementHandler (xml, xml_start, xml_end);

    XML_Parse (xml, argv[1], strlen (argv[1]), 1);
    XML_ParserFree (xml);

    free (userdata.test); /* this will crash because the previous strdup() was done with the C runtime context of libexpat and not our application */
}

If this application is linked with a different C runtime than libexpat is (which e.g can happen if one of them is debug-compiled) then this code crashes hard even though both the call to strdup() and free() is done only in our own code (all form of error checking have been ignored due to visibility).

[F.Ultra]

Wow, so many smart people here!
Can someone explain to me how Macho-o compares to elf and PE?

Depends on which context you are thinking about regarding differences. If we are talking about the very topic that is discussed in this thread then the executable format does not make any difference (i.e unnecessary linking is done at the linking stage and not due to the format of the file). The major difference between Mach-O and ELF is that Mach-O can store multiple binaries for different architectures in the same file (which ELF can do with an extension).

[F.Ultra]

Such as? (give me examples of Windows libraries btw, since you used this example against Windows -- I'm not surprised if Linux libs do have this "problem" since, well, the quality of most libraries is kind of subpar)

Of course, in Windows it's perfectly fine for a library to ask you to use one of VirtualFree, LocalFree or GlobalFree, because those are system calls and always the same no matter the runtime or dependencies (i.e. things won't break). So those don't count.

Seriously? Every library worth its salt has its own free or destroy or destructor, it's just common sense practice. This is not a corner case, this is the norm. Most of the time you don't even just "wrap free", you have to do more work to begin with when destroying the library's object.

For example, zlib, flac, etc.

If you think destroying an object via an extra function call is some sort of "serious overhead" then you shouldn't be using a dynamic library in the first place, static link it instead with LTO. That's like 0.01% of the performance hit.

Yes libraries that originate on Windows or have a large usebase on Windows have their own free() functions, not because such a function make a library better but because they are necessary on Windows while they are completely useless on every other system (unless of course if we talk about objects and opaque handlers since their destruction can be more complex than a simple call to free()).

[coder]

They're not bound to that library as evidenced by the fact that you can LD_PRELOAD a completely different library before and it steals those symbols.

That's just a trick used for testing, tracing, etc. It's not standard practice. Effectively, the symbols are bound to the libraries in which they were found at link time.

Why do people come up with such stupid arguments? Who the fuck uses multiple versions of a library from ONE code? Come on, stop grasping at straws.

We're not. It was your example that you were holding up as some sort of desirable trait. PAL and I were pointing out that it's quite hazardous, in practice.

Multiple versions can exist in the same addressing space if, say, a dependency that your program needs uses a different version of a library that your program directly accesses. But those data structs will never be messed up: your code calls version X of library and passes structs to it, while the other (deeper) dependency calls version Y of library with its own structs, and there's literally no conflict.

We know this. We're saying that when libraries at multiple levels of a software stack share a common dependency, it's likely they're passing objects from this 3rd library between them. That will usually fail pretty hard.

Also see: https://stackoverflow.com/a/33783918

... C++11 came along with mandatory changes to nearly all Standard library types.

They incremented the major version of libstdc++, for this. Again, see point about why linking different versions of the same library isn't a good idea.

Types should be local to a library. This is called encapsulation, and it's one of the first things any C or C++ programmer learns about.

This makes it pretty clear what the problem is. You're just a sad little Windows programmer pissed at the ascendancy of Linux and how it's not broken and encumbered like Windows userland.

Don't waste people's time with your bullshit.

This. While it's funny to see our resident troll fight a newcomer, it's ultimately not worth it.

Go away. We don't need you.

This forum is so full of morons who never coded a thing in their life and still talk.

If you think ELF and Linux shared libraries are so broken, maybe don't use them. No need to cry about it and insult people on here like a small child who hasn't had his nap.

[coder]

Such as? (give me examples of Windows libraries btw, since you used this example against Windows -- I'm not surprised if Linux libs do have this "problem" since, well, the quality of most libraries is kind of subpar)

Dude, this is a Linux forum. I don't know any Windows-exclusive libraries, having spent the last couple decades on Linux.

Anyway, boost, OpenCV, and Caffe are some I've been using recently that don't seem to have overloaded new/delete.

Seriously? Every library worth its salt has its own free or destroy or destructor, it's just common sense practice. This is not a corner case, this is the norm. Most of the time you don't even just "wrap free", you have to do more work to begin with when destroying the library's object.

For example, zlib, flac, etc.

Since C doesn't have destructors, it's natural to have some sort of "destroy" function. In that sense, it's common practice for C libraries.

If you think destroying an object via an extra function call is some sort of "serious overhead"

No, but I'm saying that wrapping up all of your data types in hermetically-sealed boxes, and restricting your APIs to use only POD types does. And copying data or wrapping and unwrapping as it moves up and down your software stack can quickly burn performance.