Announcement

Collapse
No announcement yet.

Fedora 29 Might Make Change To Eliminate Unnecessary Linking

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #11
    Originally posted by F.Ultra View Post
    Windows have their own hell here since you there can link with two different DLL:s that each link to different versions of a third library with all the ugliness that this can cause.
    Actually it handles this situation perfectly without conflicts, this is ugly only on Linux / Unix. Remember that symbols are not global, and in fact you don't even need symbols to import from a given DLL, you can use ordinals (indices of sorts). Clearly importing "function number 1" from a bazillion DLLs works perfectly fine, same as importing symbol ABC from a bazillion DLLs, no conflict at all, they're separate.

    Originally posted by F.Ultra View Post
    On Linux the symbols are a shared namespace per process so if your application links with say OpenSSL and you then start to use Zlib functions but forget to link with it (-lz) you can still access the exposes symbols from zlib that OpenSSL links.
    This should never be allowed. Ever. It's just stupid. Should totally give you massive error and smack you in the head for trying to do it. IMO.

    I mean on Linux/Unix shared objects, it's like using god damn a shared global variable space for the symbols, while on Windows they're all local to the DLL itself (same as on Linux if you manually use dlopen/dlsym). Even if you use 1 million versions of a library, they'll all be in separate namespaces/scopes, so each function is unique, and each of their global vars are also unique etc... no conflicts there.

    Comment


    • #12
      Originally posted by Weasel View Post
      Actually it handles this situation perfectly without conflicts, this is ugly only on Linux / Unix. Remember that symbols are not global, and in fact you don't even need symbols to import from a given DLL, you can use ordinals (indices of sorts). Clearly importing "function number 1" from a bazillion DLLs works perfectly fine, same as importing symbol ABC from a bazillion DLLs, no conflict at all, they're separate.
      It only looks perfectly without conflicts on the surface, underneath the hood all kinds of crazy is happening. Many people will probably not be exposed to it but once you use libraries where objects have to be shared between libraries all kinds of strange things can happen due to this multi-linking.

      E.g since each library can be linked with any of the gazillion libc:s that exist in Windows (4 installed in Windows per default and then each version of VS introduces 4 new ones since VS7) and memory is allocated by libc they have a context switching facility in Windows so that when your function calls a DLL, Windows switches to the libc that this DLL is linked to.

      So far so good, but let us now use something a little bit more complex, say that the shared library allocates memory (say for a url_encode() function or whatever) and you free that from your application, well then you have now freed memory that your libc never allocated and now you have made a complete mess of your applications free-memory tables. Which is why all libraries originating on Windows have a mylibrary_free() function so that Windows can do that little context switching bit.

      Then we can go a little more advanced, let's say that the library implements some form of callback functionality, say like cURL for receiveing HTTP data or libexpat for decoding XML data. You call whatever function in the shared library that initiates the transfer or file decoding and Windows switches context, however when the library calls back to your code Windows does not switch context so any memory or object that you allocate in the callbacks in your own code is now allocated with the libc that the DLL happened to be linked with. So if that version or type of libc differs from the one you use then you again mess up your free-memory lists if you ever try to free that memory/object.

      Originally posted by Weasel View Post
      This should never be allowed. Ever. It's just stupid. Should totally give you massive error and smack you in the head for trying to do it. IMO.

      I mean on Linux/Unix shared objects, it's like using god damn a shared global variable space for the symbols, while on Windows they're all local to the DLL itself (same as on Linux if you manually use dlopen/dlsym). Even if you use 1 million versions of a library, they'll all be in separate namespaces/scopes, so each function is unique, and each of their global vars are also unique etc... no conflicts there.
      Among other things this allows Linux to share memory for all libraries so that even if the same library is loaded by X number of other components in your application it still only uses the same memory.

      And it also removes the problem on Windows where you can have a new version of libXX globally in system32 but since you have an old unpatched version in the same folder where your application is this old version is loaded instead.

      Now there are benefits and drawbacks to each solution, it's just that Windows and Linux works in two completely different ways here.
      Last edited by F.Ultra; 03 July 2018, 03:04 PM.

      Comment


      • #13
        Originally posted by puleglot View Post
        Gentoo and Debian applies patch to ltmain.sh
        On debian it's not done by default

        Comment


        • #14
          Originally posted by F.Ultra View Post
          It only looks perfectly without conflicts on the surface, underneath the hood all kinds of crazy is happening. Many people will probably not be exposed to it but once you use libraries where objects have to be shared between libraries all kinds of strange things can happen due to this multi-linking.

          E.g since each library can be linked with any of the gazillion libc:s that exist in Windows (4 installed in Windows per default and then each version of VS introduces 4 new ones since VS7) and memory is allocated by libc they have a context switching facility in Windows so that when your function calls a DLL, Windows switches to the libc that this DLL is linked to.

          So far so good, but let us now use something a little bit more complex, say that the shared library allocates memory (say for a url_encode() function or whatever) and you free that from your application, well then you have now freed memory that your libc never allocated and now you have made a complete mess of your applications free-memory tables. Which is why all libraries originating on Windows have a mylibrary_free() function so that Windows can do that little context switching bit.

          Then we can go a little more advanced, let's say that the library implements some form of callback functionality, say like cURL for receiveing HTTP data or libexpat for decoding XML data. You call whatever function in the shared library that initiates the transfer or file decoding and Windows switches context, however when the library calls back to your code Windows does not switch context so any memory or object that you allocate in the callbacks in your own code is now allocated with the libc that the DLL happened to be linked with. So if that version or type of libc differs from the one you use then you again mess up your free-memory lists if you ever try to free that memory/object.
          Dude, what?

          There is literally zero context switches between libraries because they're in the same address space (unless you mean different threads but that's a completely different thing). That's the whole point of libraries compared to processes in the first place. I have seriously NO idea where you got that from but it's so wrong it's not even funny.

          Yes, if you use malloc from one library and then pass this pointer and expect another library (or the app itself) to free it, it's your own damn fault because it WILL break.

          Proper design is to give opaque pointers/handles to the app in question and when it needs to free it, pass it back to your library which will free it. For example, let's say your library has a "CreateNewFoobar" function which calls its own malloc and returns the pointer. The app then should call the library's "FreeFoobar" function and there's literally zero problems, not call "free" on it by itself, that's just retarded.

          Imagine if the library uses its own, internal malloc, how stupid can it be to use a generic free() on it when no data structure will match? The point is to treat the library's allocations as a "black box" so freeing something it allocated is beyond stupid.

          Rule of Thumb: Never let another module free what you have allocated or manipulate your objects without using your library's APIs. That's what they're for.

          Originally posted by F.Ultra View Post
          Among other things this allows Linux to share memory for all libraries so that even if the same library is loaded by X number of other components in your application it still only uses the same memory.
          This is bogus and has nothing to do with symbol lookup. Windows shares memory just fine. Pages that are identical get shared.

          32-bit DLLs with relocations might not be shared if they're loaded in different ranges since the code has to be patched (no relative addressing) so the pages won't be identical anymore. This is implemented with Copy-On-Write at the OS level, just like Linux uses it when doing a fork() otherwise it would be painfully slow to launch a new process on Linux.

          But it has nothing whatsoever to do with symbols and they both share in the same way.

          Originally posted by F.Ultra View Post
          And it also removes the problem on Windows where you can have a new version of libXX globally in system32 but since you have an old unpatched version in the same folder where your application is this old version is loaded instead.
          This has nothing to do with this whatsoever. That's purely the Windows loader's problem, because it looks for apps in the same directory as the app first. Nothing about the DLL format forces this to happen, in fact that happens before the DLL is even read or parsed.

          Which has nothing to do with the fact that DLLs look up symbols by module (each symbol is in a subtable of a module) so it is technically *impossible* to import a symbol without a module.


          Sorry but you really have not the slightest clue of what you're talking about... as proof of this Wine doesn't suffer from a single thing you've stated, and yet Wine has superior symbol lookup than ELF since it implements a DLL loader.
          Last edited by Weasel; 03 July 2018, 05:12 PM.

          Comment


          • #15
            Originally posted by timofonic View Post
            I'm extremely surprised this isn't yet done by ALL major distributions...
            Actually, looks like Arch Linux makepkg.conf includes it by default.

            Comment


            • #16
              Originally posted by Weasel View Post
              I mean on Linux/Unix shared objects, it's like using god damn a shared global variable space for the symbols, while on Windows they're all local to the DLL itself (same as on Linux if you manually use dlopen/dlsym). Even if you use 1 million versions of a library, they'll all be in separate namespaces/scopes, so each function is unique, and each of their global vars are also unique etc... no conflicts there.
              Don't bash what you clearly don't understand.

              Shared library symbol references are only global if they're not resolved. If you link to a library providing those symbols, then they're now bound to that library. If every library links in its immediate dependencies, like everyone here seems to agree they should, then there's no problem.

              To avoid symbol collisions, people use techniques like unique prefixes or C++ namespaces.

              Now, try using "1 million versions of a library" in a single executable and pass around data structures between different users of the different versions of those libraries and have it not blow up in your face, please. Can't do it? Gee, then maybe linking in different versions of the same shared library is a bad idea! ...not that you can't do it, on properly versioned shared libraries, but people generally don't because it's so perilous.

              Comment


              • #17
                Originally posted by Weasel View Post
                The stupidity in Linux userland never ceases to amaze me.

                "Import a symbol from a library but don't link to it", WTF? That's not even possible on Windows DLLs since symbols are part of the imports from a library. And rightfully so.
                The stupidity of phoronix commenters never ceases to amaze me.
                it is the other way around: by default library is linked in even when no symbols are needed

                Comment


                • #18
                  Originally posted by Weasel View Post
                  This should never be allowed. Ever. It's just stupid. Should totally give you massive error and smack you in the head for trying to do it. IMO.
                  you shouldn't be allowed to post your bullshit ideas. ever. it's just stupid
                  Originally posted by Weasel View Post
                  I mean on Linux/Unix shared objects, it's like using god damn a shared global variable space for the symbols, while on Windows they're all local to the DLL itself (same as on Linux if you manually use dlopen/dlsym). Even if you use 1 million versions of a library, they'll all be in separate namespaces/scopes, so each function is unique, and each of their global vars are also unique etc... no conflicts there.
                  what a wall of shit
                  you described a world where symbol interposition does not exist, i.e. no custom malloc replacements or no ld_preload
                  furthermore, having different versions of library in same process is very dangerous
                  because then you can pass pointer to data created by one version to function from other version and it can expect its private data in different format. i.e. crash. or it can have its global data inconsistent with existence of object created by someone else. and again crash
                  even windows compiler has option for selecting static or shared runtime. static is exactly as you like: everyone gets his own copy. and rule number one of windows dll programming: always use shared runtime
                  Last edited by pal666; 03 July 2018, 09:58 PM.

                  Comment


                  • #19
                    Originally posted by Weasel View Post
                    Yes, if you use malloc from one library and then pass this pointer and expect another library (or the app itself) to free it, it's your own damn fault because it WILL break.
                    imbecile, consider your library takes const std :: string & parameter
                    who the fuck is going to call its constructor, which does allocation ?
                    and same example with non-const reference?
                    all std::string's allocations and deallocations are "black box" and automatic at that, i.e. you don't have to remember to call something, but this black box breaks when there are more than one c++ standard library is linked into process
                    Originally posted by Weasel View Post
                    Proper design is to give opaque pointers/handles to the app in question and when it needs to free it, pass it back to your library which will free it
                    this is pile of shit to workaround windows deficiency and it will not work anyway.
                    consider your handle from liba is for some large object which has field of some other type governed by libb.
                    you have api liba_make_handle and liba_get_b_from_handle
                    now you've got your b object created by liba's libb, not by your libb. how the fuck you are going to access it? should liba provide copy of all libb api as liba_libb_doxxx ?
                    Originally posted by Weasel View Post
                    This is bogus and has nothing to do with symbol lookup. Windows shares memory just fine. Pages that are identical get shared.
                    pages of different files are not identical
                    Originally posted by Weasel View Post
                    32-bit DLLs with relocations might not be shared if they're loaded in different ranges since the code has to be patched (no relative addressing) so the pages won't be identical anymore. This is implemented with Copy-On-Write at the OS level, just like Linux uses it when doing a fork() otherwise it would be painfully slow to launch a new process on Linux.
                    it has nothing to do with fork. it is an example of windows not supporting position independent code. linux does same thing when compiled without -fpic/-fpie. but you compile relocatable code properly and all relocations are in one contiguous array, so only this array is cowed
                    and to quote you:
                    Sorry but you really have not the slightest clue of what you're talking about
                    Last edited by pal666; 03 July 2018, 10:20 PM.

                    Comment


                    • #20
                      Originally posted by Weasel View Post
                      Yes, if you use malloc from one library and then pass this pointer and expect another library (or the app itself) to free it, it's your own damn fault because it WILL break.

                      Proper design is to give opaque pointers/handles to the app in question and when it needs to free it, pass it back to your library which will free it. For example, let's say your library has a "CreateNewFoobar" function which calls its own malloc and returns the pointer. The app then should call the library's "FreeFoobar" function and there's literally zero problems, not call "free" on it by itself, that's just retarded.

                      Imagine if the library uses its own, internal malloc, how stupid can it be to use a generic free() on it when no data structure will match? The point is to treat the library's allocations as a "black box" so freeing something it allocated is beyond stupid.

                      Rule of Thumb: Never let another module free what you have allocated or manipulate your objects without using your library's APIs. That's what they're for.
                      I think you've got the causality wrong. You're accepting Windows' constraint as given and then looking to justify it.

                      I've been down the road of custom allocators, a few times, and never found them justified. Especially for API-public types. But that's not even what we're talking about. We're talking about the exact opposite - a separate allocator for each library, by default, even though 99% of the time it's just the same underlying implementation. Were I to match your hyperbole, I could use some choice words to describe that architecture.

                      Comment

                      Working...
                      X