Announcement

Collapse
No announcement yet.

Fedora 29 Might Make Change To Eliminate Unnecessary Linking

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #11
    Originally posted by Weasel View Post
    Actually it handles this situation perfectly without conflicts, this is ugly only on Linux / Unix. Remember that symbols are not global, and in fact you don't even need symbols to import from a given DLL, you can use ordinals (indices of sorts). Clearly importing "function number 1" from a bazillion DLLs works perfectly fine, same as importing symbol ABC from a bazillion DLLs, no conflict at all, they're separate.
    It only looks perfectly without conflicts on the surface, underneath the hood all kinds of crazy is happening. Many people will probably not be exposed to it but once you use libraries where objects have to be shared between libraries all kinds of strange things can happen due to this multi-linking.

    E.g since each library can be linked with any of the gazillion libc:s that exist in Windows (4 installed in Windows per default and then each version of VS introduces 4 new ones since VS7) and memory is allocated by libc they have a context switching facility in Windows so that when your function calls a DLL, Windows switches to the libc that this DLL is linked to.

    So far so good, but let us now use something a little bit more complex, say that the shared library allocates memory (say for a url_encode() function or whatever) and you free that from your application, well then you have now freed memory that your libc never allocated and now you have made a complete mess of your applications free-memory tables. Which is why all libraries originating on Windows have a mylibrary_free() function so that Windows can do that little context switching bit.

    Then we can go a little more advanced, let's say that the library implements some form of callback functionality, say like cURL for receiveing HTTP data or libexpat for decoding XML data. You call whatever function in the shared library that initiates the transfer or file decoding and Windows switches context, however when the library calls back to your code Windows does not switch context so any memory or object that you allocate in the callbacks in your own code is now allocated with the libc that the DLL happened to be linked with. So if that version or type of libc differs from the one you use then you again mess up your free-memory lists if you ever try to free that memory/object.

    Originally posted by Weasel View Post
    This should never be allowed. Ever. It's just stupid. Should totally give you massive error and smack you in the head for trying to do it. IMO.

    I mean on Linux/Unix shared objects, it's like using god damn a shared global variable space for the symbols, while on Windows they're all local to the DLL itself (same as on Linux if you manually use dlopen/dlsym). Even if you use 1 million versions of a library, they'll all be in separate namespaces/scopes, so each function is unique, and each of their global vars are also unique etc... no conflicts there.
    Among other things this allows Linux to share memory for all libraries so that even if the same library is loaded by X number of other components in your application it still only uses the same memory.

    And it also removes the problem on Windows where you can have a new version of libXX globally in system32 but since you have an old unpatched version in the same folder where your application is this old version is loaded instead.

    Now there are benefits and drawbacks to each solution, it's just that Windows and Linux works in two completely different ways here.
    Last edited by F.Ultra; 03 July 2018, 03:04 PM.

    Comment


    • #12
      Originally posted by puleglot View Post
      Gentoo and Debian applies patch to ltmain.sh
      On debian it's not done by default

      Comment


      • #13
        Originally posted by F.Ultra View Post
        It only looks perfectly without conflicts on the surface, underneath the hood all kinds of crazy is happening. Many people will probably not be exposed to it but once you use libraries where objects have to be shared between libraries all kinds of strange things can happen due to this multi-linking.

        E.g since each library can be linked with any of the gazillion libc:s that exist in Windows (4 installed in Windows per default and then each version of VS introduces 4 new ones since VS7) and memory is allocated by libc they have a context switching facility in Windows so that when your function calls a DLL, Windows switches to the libc that this DLL is linked to.

        So far so good, but let us now use something a little bit more complex, say that the shared library allocates memory (say for a url_encode() function or whatever) and you free that from your application, well then you have now freed memory that your libc never allocated and now you have made a complete mess of your applications free-memory tables. Which is why all libraries originating on Windows have a mylibrary_free() function so that Windows can do that little context switching bit.

        Then we can go a little more advanced, let's say that the library implements some form of callback functionality, say like cURL for receiveing HTTP data or libexpat for decoding XML data. You call whatever function in the shared library that initiates the transfer or file decoding and Windows switches context, however when the library calls back to your code Windows does not switch context so any memory or object that you allocate in the callbacks in your own code is now allocated with the libc that the DLL happened to be linked with. So if that version or type of libc differs from the one you use then you again mess up your free-memory lists if you ever try to free that memory/object.
        Dude, what?

        There is literally zero context switches between libraries because they're in the same address space (unless you mean different threads but that's a completely different thing). That's the whole point of libraries compared to processes in the first place. I have seriously NO idea where you got that from but it's so wrong it's not even funny.

        Yes, if you use malloc from one library and then pass this pointer and expect another library (or the app itself) to free it, it's your own damn fault because it WILL break.

        Proper design is to give opaque pointers/handles to the app in question and when it needs to free it, pass it back to your library which will free it. For example, let's say your library has a "CreateNewFoobar" function which calls its own malloc and returns the pointer. The app then should call the library's "FreeFoobar" function and there's literally zero problems, not call "free" on it by itself, that's just retarded.

        Imagine if the library uses its own, internal malloc, how stupid can it be to use a generic free() on it when no data structure will match? The point is to treat the library's allocations as a "black box" so freeing something it allocated is beyond stupid.

        Rule of Thumb: Never let another module free what you have allocated or manipulate your objects without using your library's APIs. That's what they're for.

        Originally posted by F.Ultra View Post
        Among other things this allows Linux to share memory for all libraries so that even if the same library is loaded by X number of other components in your application it still only uses the same memory.
        This is bogus and has nothing to do with symbol lookup. Windows shares memory just fine. Pages that are identical get shared.

        32-bit DLLs with relocations might not be shared if they're loaded in different ranges since the code has to be patched (no relative addressing) so the pages won't be identical anymore. This is implemented with Copy-On-Write at the OS level, just like Linux uses it when doing a fork() otherwise it would be painfully slow to launch a new process on Linux.

        But it has nothing whatsoever to do with symbols and they both share in the same way.

        Originally posted by F.Ultra View Post
        And it also removes the problem on Windows where you can have a new version of libXX globally in system32 but since you have an old unpatched version in the same folder where your application is this old version is loaded instead.
        This has nothing to do with this whatsoever. That's purely the Windows loader's problem, because it looks for apps in the same directory as the app first. Nothing about the DLL format forces this to happen, in fact that happens before the DLL is even read or parsed.

        Which has nothing to do with the fact that DLLs look up symbols by module (each symbol is in a subtable of a module) so it is technically *impossible* to import a symbol without a module.


        Sorry but you really have not the slightest clue of what you're talking about... as proof of this Wine doesn't suffer from a single thing you've stated, and yet Wine has superior symbol lookup than ELF since it implements a DLL loader.
        Last edited by Weasel; 03 July 2018, 05:12 PM.

        Comment


        • #14
          Originally posted by timofonic View Post
          I'm extremely surprised this isn't yet done by ALL major distributions...
          Actually, looks like Arch Linux makepkg.conf includes it by default.

          Comment


          • #15
            Originally posted by Weasel View Post
            I mean on Linux/Unix shared objects, it's like using god damn a shared global variable space for the symbols, while on Windows they're all local to the DLL itself (same as on Linux if you manually use dlopen/dlsym). Even if you use 1 million versions of a library, they'll all be in separate namespaces/scopes, so each function is unique, and each of their global vars are also unique etc... no conflicts there.
            Don't bash what you clearly don't understand.

            Shared library symbol references are only global if they're not resolved. If you link to a library providing those symbols, then they're now bound to that library. If every library links in its immediate dependencies, like everyone here seems to agree they should, then there's no problem.

            To avoid symbol collisions, people use techniques like unique prefixes or C++ namespaces.

            Now, try using "1 million versions of a library" in a single executable and pass around data structures between different users of the different versions of those libraries and have it not blow up in your face, please. Can't do it? Gee, then maybe linking in different versions of the same shared library is a bad idea! ...not that you can't do it, on properly versioned shared libraries, but people generally don't because it's so perilous.

            Comment


            • #16
              Originally posted by Weasel View Post
              The stupidity in Linux userland never ceases to amaze me.

              "Import a symbol from a library but don't link to it", WTF? That's not even possible on Windows DLLs since symbols are part of the imports from a library. And rightfully so.
              The stupidity of phoronix commenters never ceases to amaze me.
              it is the other way around: by default library is linked in even when no symbols are needed

              Comment


              • #17
                Originally posted by Weasel View Post
                This should never be allowed. Ever. It's just stupid. Should totally give you massive error and smack you in the head for trying to do it. IMO.
                you shouldn't be allowed to post your bullshit ideas. ever. it's just stupid
                Originally posted by Weasel View Post
                I mean on Linux/Unix shared objects, it's like using god damn a shared global variable space for the symbols, while on Windows they're all local to the DLL itself (same as on Linux if you manually use dlopen/dlsym). Even if you use 1 million versions of a library, they'll all be in separate namespaces/scopes, so each function is unique, and each of their global vars are also unique etc... no conflicts there.
                what a wall of shit
                you described a world where symbol interposition does not exist, i.e. no custom malloc replacements or no ld_preload
                furthermore, having different versions of library in same process is very dangerous
                because then you can pass pointer to data created by one version to function from other version and it can expect its private data in different format. i.e. crash. or it can have its global data inconsistent with existence of object created by someone else. and again crash
                even windows compiler has option for selecting static or shared runtime. static is exactly as you like: everyone gets his own copy. and rule number one of windows dll programming: always use shared runtime
                Last edited by pal666; 03 July 2018, 09:58 PM.

                Comment


                • #18
                  Originally posted by Weasel View Post
                  Yes, if you use malloc from one library and then pass this pointer and expect another library (or the app itself) to free it, it's your own damn fault because it WILL break.
                  imbecile, consider your library takes const std :: string & parameter
                  who the fuck is going to call its constructor, which does allocation ?
                  and same example with non-const reference?
                  all std::string's allocations and deallocations are "black box" and automatic at that, i.e. you don't have to remember to call something, but this black box breaks when there are more than one c++ standard library is linked into process
                  Originally posted by Weasel View Post
                  Proper design is to give opaque pointers/handles to the app in question and when it needs to free it, pass it back to your library which will free it
                  this is pile of shit to workaround windows deficiency and it will not work anyway.
                  consider your handle from liba is for some large object which has field of some other type governed by libb.
                  you have api liba_make_handle and liba_get_b_from_handle
                  now you've got your b object created by liba's libb, not by your libb. how the fuck you are going to access it? should liba provide copy of all libb api as liba_libb_doxxx ?
                  Originally posted by Weasel View Post
                  This is bogus and has nothing to do with symbol lookup. Windows shares memory just fine. Pages that are identical get shared.
                  pages of different files are not identical
                  Originally posted by Weasel View Post
                  32-bit DLLs with relocations might not be shared if they're loaded in different ranges since the code has to be patched (no relative addressing) so the pages won't be identical anymore. This is implemented with Copy-On-Write at the OS level, just like Linux uses it when doing a fork() otherwise it would be painfully slow to launch a new process on Linux.
                  it has nothing to do with fork. it is an example of windows not supporting position independent code. linux does same thing when compiled without -fpic/-fpie. but you compile relocatable code properly and all relocations are in one contiguous array, so only this array is cowed
                  and to quote you:
                  Sorry but you really have not the slightest clue of what you're talking about
                  Last edited by pal666; 03 July 2018, 10:20 PM.

                  Comment


                  • #19
                    Originally posted by Weasel View Post
                    Yes, if you use malloc from one library and then pass this pointer and expect another library (or the app itself) to free it, it's your own damn fault because it WILL break.

                    Proper design is to give opaque pointers/handles to the app in question and when it needs to free it, pass it back to your library which will free it. For example, let's say your library has a "CreateNewFoobar" function which calls its own malloc and returns the pointer. The app then should call the library's "FreeFoobar" function and there's literally zero problems, not call "free" on it by itself, that's just retarded.

                    Imagine if the library uses its own, internal malloc, how stupid can it be to use a generic free() on it when no data structure will match? The point is to treat the library's allocations as a "black box" so freeing something it allocated is beyond stupid.

                    Rule of Thumb: Never let another module free what you have allocated or manipulate your objects without using your library's APIs. That's what they're for.
                    I think you've got the causality wrong. You're accepting Windows' constraint as given and then looking to justify it.

                    I've been down the road of custom allocators, a few times, and never found them justified. Especially for API-public types. But that's not even what we're talking about. We're talking about the exact opposite - a separate allocator for each library, by default, even though 99% of the time it's just the same underlying implementation. Were I to match your hyperbole, I could use some choice words to describe that architecture.

                    Comment


                    • #20
                      Originally posted by coder View Post
                      Don't bash what you clearly don't understand.

                      Shared library symbol references are only global if they're not resolved. If you link to a library providing those symbols, then they're now bound to that library. If every library links in its immediate dependencies, like everyone here seems to agree they should, then there's no problem.
                      They're not bound to that library as evidenced by the fact that you can LD_PRELOAD a completely different library before and it steals those symbols. Obviously, once a symbol is resolved, it won't resolve again, that doesn't mean it's "bound" to a library since this can be changed on every load of the library via LD_PRELOAD. Bound means that it is bound AT COMPILATION TIME.

                      Originally posted by coder View Post
                      Now, try using "1 million versions of a library" in a single executable and pass around data structures between different users of the different versions of those libraries and have it not blow up in your face, please. Can't do it? Gee, then maybe linking in different versions of the same shared library is a bad idea! ...not that you can't do it, on properly versioned shared libraries, but people generally don't because it's so perilous.
                      Why do people come up with such stupid arguments? Who the fuck uses multiple versions of a library from ONE code? Come on, stop grasping at straws.

                      Multiple versions can exist in the same addressing space if, say, a dependency that your program needs uses a different version of a library that your program directly accesses. But those data structs will never be messed up: your code calls version X of library and passes structs to it, while the other (deeper) dependency calls version Y of library with its own structs, and there's literally no conflict.

                      Originally posted by pal666 View Post
                      The stupidity of phoronix commenters never ceases to amaze me.
                      it is the other way around: by default library is linked in even when no symbols are needed
                      Took the words right out of my mouth with the first phrase there. Not what the guy I quoted said. Maybe keep context next time unless you want to look like an idiot.

                      He was outlining why removing such "dead links" to libraries can lead to breakage, but this only happens due to the stupidity in Linux userland.

                      Originally posted by pal666 View Post
                      what a wall of shit
                      you described a world where symbol interposition does not exist, i.e. no custom malloc replacements or no ld_preload
                      Good because LD_PRELOAD is a hack that shouldn't exist in the first place.

                      Originally posted by pal666 View Post
                      imbecile, consider your library takes const std :: string & parameter
                      who the fuck is going to call its constructor, which does allocation ?
                      Hey moron, did you know if your library takes non-basic built-in types, you're a retarded programmer? Standard library is just a collection of templated classes, so they're far from "basic types".

                      Go back to school. I'm not interested in "what ifs" from code developed by rookies.

                      Here, educate yourself: https://stackoverflow.com/a/22797419

                      Also see: https://stackoverflow.com/a/33783918
                      Originally posted by Ben Voigt
                      This is something that Linux programmers weren't accustomed to dealing with, because g++'s libstdc++ was a de-facto standard and virtually all programs used it, thus satisfying the ODR. clang's libc++ broke that assumption, and then C++11 came along with mandatory changes to nearly all Standard library types.

                      Just don't share Standard library types between modules. It's undefined behavior.
                      I'm guessing you don't even know what "undefined behavior" is.


                      ________________

                      Originally posted by pal666 View Post
                      this is pile of shit to workaround windows deficiency and it will not work anyway.
                      consider your handle from liba is for some large object which has field of some other type governed by libb.
                      Types should be local to a library. This is called encapsulation, and it's one of the first things any C or C++ programmer learns about.

                      What the fuck kind of spagetti bullshit do you describe? In C++, a class should implement all associated interfaces within itself, or free functions within the same namespace (for ADL lookup). Two libraries must NEVER share namespace.

                      If liba can't handle everything it can with its type then it's time to redesign your shitty liba.

                      Originally posted by pal666 View Post
                      pages of different files are not identical
                      Thanks for the info Captain Obvious. We're talking about sharing the same library in memory. Same library that, you know, gets mapped as COW.

                      Originally posted by pal666 View Post
                      it has nothing to do with fork. it is an example of windows not supporting position independent code.
                      LMFAO what has Windows to do with the CPU? Are you saying Windows can't use the CPU's rip register in 64-bit mode?

                      Don't waste people's time with your bullshit.

                      https://en.wikipedia.org/wiki/Positi...e#Windows_DLLs
                      Originally posted by Wikipedia
                      In Windows Vista and later versions of Windows, the relocation of DLLs and executables is done by the kernel memory manager, which shares the relocated binaries across multiple processes. Images are always relocated from their preferred base addresses, achieving address space layout randomization (ASLR).[4]
                      This forum is so full of morons who never coded a thing in their life and still talk.
                      Last edited by Weasel; 04 July 2018, 07:42 AM.

                      Comment

                      Working...
                      X