Fedora Stakeholders Debate Statically Linking Python For Better Performance

[coder]

Mostly it is somehow "burried" as Linear Algebra, Math etc Lib on your system like BLAS, ARPACK, FFTW, LAPACK, ACML, MKL ...what so ever. But each modern "research-friendly" Language or Environment is using it.

FFTW does not use it. ACML and MKL aren't even in my distro. Things like LAPACK tend to get replaced with more modern & language-native libraries, like Eigen.

And, as I said in the above post, implementations based on new technology do not use it, such as cuBLAS and MKL-DNN.

This stuff for sure is not dead - have a look at e.g. OpenBLAS, GraphBLAS or NVBLAS. It is still under development and adjusted to the latest CPU architectures, "ported" as GPU offload, or reimplemented in new algorithms.

You're just citing the long tail of the curve. Until every single one of those stragglers dies off (which they won't, as long as virtually anything still depends on them), people like you will keep claiming it's not dead.

Well, friend, the parrot might not be dead, but he's far from lively.

[coder]

But does a mature language need a lot of new code?

Well, if you figure the old code is continually getting replaced (probably according to some sort of half-life function), then a language that's no longer used for new code will eventually die off. An enterprising individual could plot the number of fortran dependencies in some distro with a periodic release cycle. It'd be interesting to see if it indeed followed an exponential decay.

As for why it tends to get replaced, consider that the type of numerical code for which fortran-based libraries are still used is typically worth optimizing to use SIMD instruction set extensions, heavy multi-threading, or even GP GPU. While you can do all of those things in fortran (see OpenMP, OpenACC), there are sometimes big benefits to be had from hand-crafting an implementation for a given technology. Otherwise, why aren't most people still using the original BLAS or fortran-based FFT implementations?

[CochainComplex]

Yes, in decades past.

These days, fortran seems to be a direct dependency only of older libraries, with not much new code being written in it.

Mostly it is somehow "burried" as Linear Algebra, Math etc Lib on your system like BLAS, ARPACK, FFTW, LAPACK, ACML, MKL ...what so ever. But each modern "research-friendly" Language or Environment is using it.
This stuff for sure is not dead - have a look at e.g. OpenBLAS, GraphBLAS or NVBLAS. It is still under development and adjusted to the latest CPU architectures, "ported" as GPU offload, or reimplemented in new algorithms.

[ssokolow]

Idiomatic Go does not even depend on libc. Go encourages you to not use libc, and linking against it is disabled by default when you cross-compile.
You are correct regarding how other dependencies work in Go. All dependencies are statically linked into one fat binary together with the supplied Go-runtime. Another thing to note about dependencies in Go projects, is that they very often end up picking a plethora of random libraries from east and west on Github. As an example, consider the dependency list for Kubernetes. This is also partly encouraged by the Go eco-system and toolchain.

...but, when that "abstract away the platform so thoroughly that we're not even depending on libc" approach leaks, it leaks hard, resulting in blog posts like this:

https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/

This is a pythons strong point I think everybody understands thAt! The problem with that attitude is that new technologies get ignored and struggle to build that infrastructure. It is pretty easy to see the advantages of new languages like Swift, Julia and others but if they aren’t used and don’t have Pythons infrastructure they will never grow into Python replacements.

frankly I do believe a Python replacement is needed. The lack of any potential for performance out of arbitrary code is a big factor that will never be overcome. Other issues like a sound distribution model (maybe static linking will help here) are also hold backs.

in any event the point here is that if new technologies aren’t explored then we will never gain new solutions.

Point. In hindsight, I was assuming perfect knowledge of the alternatives in imputing a motivation for choosing Python.

[starshipeleven]

But does a mature language need a lot of new code?

I'm not sure what you mean here. People still write a ton of C (or C-like code) even in 2019, and C is very mature.

If we are talking about research, then each experiment or model or test they do they write new code to make the system do different calculations (different expressions, matrix, whatever). Or at least it is much more likely that they will need to change something for the next test or experiment.

It's different from businness (classic example is banks) where the core logic does not really change in decades because they keep doing the same thing over and over and over, so you still have ancient codebases in Cobol.

[wizard69]

You might as well build it into every binary if the future is to ship each binary with its own set of dynamic libraries via snap/flatpak.

Isn’t that the truth! Effectively you have statically linked dynamic libs with these systems.

[wizard69]

Yes. Unless they forget to do that on each and every statically linked package. Or are you implying packages and build scripts are always perfect? That's the whole stupidity of redundancy. It's just one more thing you always need to take care of and remember. Have you heard of database normalization? The whole point of that is the elimination of as much redundancy as possible. Why? Because storing the SAME information in several places is friggin STUPID and can lead to nothing but problems in the future. If you have to do that, you're most definitely doing something wrong.

Because applying the same change to 2 binaries is stupid when you could do that to just one. And it scales to eternity with the number of static links you have. Why on Earth would you do that? Why wouldn't you fix the actual problem instead, that is, improve dynaload performance? The whole world started using dynamic linking because the performance impact was proved to be negligible. How is it that Python still manages to fail in this regard? Surely it's not their fault, and the whole world is wrong instead?

You seem to be thinking from the big systems perspective and thus you might be biased to a certain perspective. Consider an embedded or more limited system it can actually be more work to keep a limited number of apps in sync with each library update. This especially if rigorous validations are required. Sometimes it makes more sense to only update what is known to have issues instead of testing everything that might be linked to a shared library.

your approach has its own problems as you can’t know for sure how every app in a large system will behave with a shared library update. This can force updates to apps that might not need it otherwise producing unhappy users. I’m not saying shared libraries are wrong just that sometimes it makes sense to statically link a library. The program using that library does not impact other users of the dynamically linked solution and that program is not impacted by arbitrary system updates.

neither a statically linked solution nor a dynamically linked one is a perfect solution. However each solution has its strengths and in the case of Python it might be the right path to take to statically link the lib.

[wizard69]

Do you think bundling libssl, gnutls, glibc, curl etc into each application would be wise?

You only bundle what you need!

even then the value of a statically linked program depends upon that program. Value would range from not useful at all to solving a bunch of problems. In the context of Python, static linking might actually solve a bunch of problems or at least lead to better solutions.

[wizard69]

The benchmarks game results should speak for themselves.



No matter what you're trying to do with a language, "existing ecosystem" tends to be the reason people stick to their existing sub-optimal choices. (eg. I'd love to work exclusively in Rust, but it has no mature Qt bindings or robust, semi-autogenerated ORM migrations or Django ecosystem or mature OpenCV bindings.)

This is a pythons strong point I think everybody understands thAt! The problem with that attitude is that new technologies get ignored and struggle to build that infrastructure. It is pretty easy to see the advantages of new languages like Swift, Julia and others but if they aren’t used and don’t have Pythons infrastructure they will never grow into Python replacements.

frankly I do believe a Python replacement is needed. The lack of any potential for performance out of arbitrary code is a big factor that will never be overcome. Other issues like a sound distribution model (maybe static linking will help here) are also hold backs.

in any event the point here is that if new technologies aren’t explored then we will never gain new solutions.

[Vistaus]

Yes, in decades past.

These days, fortran seems to be a direct dependency only of older libraries, with not much new code being written in it.

But does a mature language need a lot of new code?