Fedora Stakeholders Debate Statically Linking Python For Better Performance

[ermo]

Ah yes, I was sure someone will imply that if redundancy is bad on Windows, it must be because Windows suxx, therefore Windows developers are idiots. And Linux is awesome, therefore Linux developers are geniuses, therefore, the lessons learned on Windows are somehow irrelevant and Linux apps are somehow immune to the inherent problems of static linking.

If you still can't tell if that was sarcasm or not, let me help you out: GDI and msvcrt.dll were just metaphores and examples of the underlying problem.

Regarding Go, I haven't really looked into it, if they do that, I retract my statement about that.

All I said was that, since Go appears to do the things you warn against, how is Go a good example compared to statically linking Python (which you appear to warn against by specifically referencing a specific MS GDI+ example)?

I also never implied that Windows sucks nor that Windows developers are idiots etc. That part is all on you. I thought it obvious that the software development part of the IT community as such should learn from the bad old Windows days of bundling DLLs willy nilly without bothering to update them even after serious security vulnerabilities become known.

The only thing 'the bad old days of MS software development' was supposed to refer to is the fact that software distribution on Windows (outside of MS controlled code bases) never really had the kind of central repositories that the Linux community now takes for granted (then again, 2004 was 15 years ago. Internet access has become much more prevalent since then). This has traditionally made it impossible to reap the benefits of dynamic linking on a repository-wide basis for Windows developers.

These days, the closest equivalent to a central Linux distribution repository for Windows is probably chocolatey (which is maintained by a 3rd party)?

[Delgarde]

There are other programs, which use libpython, i.e. some $WINDOW_MANAGER, which uses it for internal scripting stuff. They will possibly get statically linked, too.

And that's the complicating factor. The controversy of the proposal isn't really over disk space, it's over the consequences of the change throughout the system. Nobody really cares if Python itself statically links to it's own libraries... but there's a _lot_ of stuff in a modern distro which links to libpython to provide scripting functionality. Lots of risk, balanced by a benchmark performance gain that may or may not benefit real usage.

[oleid]

And that's the complicating factor. The controversy of the proposal isn't really over disk space, it's over the consequences of the change throughout the system. Nobody really cares if Python itself statically links to it's own libraries... but there's a _lot_ of stuff in a modern distro which links to libpython to provide scripting functionality. Lots of risk, balanced by a benchmark performance gain that may or may not benefit real usage.

Maybe the interpreter should be statistically linked and the rest use a dynamic version of libpython?

[pal666]

Bash is not used for actual productive software and scientific work (not as the main payload logic anyway)

i'm pretty sure productive and scientific python is thin wrapper around real implementation in faster language, i.e. it will not benefit from subj at all

[starshipeleven]

i'm pretty sure productive and scientific python is thin wrapper around real implementation in faster language, i.e. it will not benefit from subj at all

You are pretty wrong. There is plenty of small and medium research projects done with Python. Same story as with Java (Matlab), the researchers aren't true programmers.

[Vistaus]

Yes, in decades past.

These days, fortran seems to be a direct dependency only of older libraries, with not much new code being written in it.

But does a mature language need a lot of new code?

[wizard69]

The benchmarks game results should speak for themselves.



No matter what you're trying to do with a language, "existing ecosystem" tends to be the reason people stick to their existing sub-optimal choices. (eg. I'd love to work exclusively in Rust, but it has no mature Qt bindings or robust, semi-autogenerated ORM migrations or Django ecosystem or mature OpenCV bindings.)

This is a pythons strong point I think everybody understands thAt! The problem with that attitude is that new technologies get ignored and struggle to build that infrastructure. It is pretty easy to see the advantages of new languages like Swift, Julia and others but if they aren’t used and don’t have Pythons infrastructure they will never grow into Python replacements.

frankly I do believe a Python replacement is needed. The lack of any potential for performance out of arbitrary code is a big factor that will never be overcome. Other issues like a sound distribution model (maybe static linking will help here) are also hold backs.

in any event the point here is that if new technologies aren’t explored then we will never gain new solutions.

[wizard69]

Do you think bundling libssl, gnutls, glibc, curl etc into each application would be wise?

You only bundle what you need!

even then the value of a statically linked program depends upon that program. Value would range from not useful at all to solving a bunch of problems. In the context of Python, static linking might actually solve a bunch of problems or at least lead to better solutions.

[wizard69]

Yes. Unless they forget to do that on each and every statically linked package. Or are you implying packages and build scripts are always perfect? That's the whole stupidity of redundancy. It's just one more thing you always need to take care of and remember. Have you heard of database normalization? The whole point of that is the elimination of as much redundancy as possible. Why? Because storing the SAME information in several places is friggin STUPID and can lead to nothing but problems in the future. If you have to do that, you're most definitely doing something wrong.

Because applying the same change to 2 binaries is stupid when you could do that to just one. And it scales to eternity with the number of static links you have. Why on Earth would you do that? Why wouldn't you fix the actual problem instead, that is, improve dynaload performance? The whole world started using dynamic linking because the performance impact was proved to be negligible. How is it that Python still manages to fail in this regard? Surely it's not their fault, and the whole world is wrong instead?

You seem to be thinking from the big systems perspective and thus you might be biased to a certain perspective. Consider an embedded or more limited system it can actually be more work to keep a limited number of apps in sync with each library update. This especially if rigorous validations are required. Sometimes it makes more sense to only update what is known to have issues instead of testing everything that might be linked to a shared library.

your approach has its own problems as you can’t know for sure how every app in a large system will behave with a shared library update. This can force updates to apps that might not need it otherwise producing unhappy users. I’m not saying shared libraries are wrong just that sometimes it makes sense to statically link a library. The program using that library does not impact other users of the dynamically linked solution and that program is not impacted by arbitrary system updates.

neither a statically linked solution nor a dynamically linked one is a perfect solution. However each solution has its strengths and in the case of Python it might be the right path to take to statically link the lib.

[wizard69]

You might as well build it into every binary if the future is to ship each binary with its own set of dynamic libraries via snap/flatpak.

Isn’t that the truth! Effectively you have statically linked dynamic libs with these systems.