Fedora Stakeholders Debate Statically Linking Python For Better Performance

[phoronix]

Phoronix: Fedora Stakeholders Debate Statically Linking Python For Better Performance

A surprisingly controversial proposal for Fedora 32 is to shift from dynamically linking Python 3 with the libpython3.X.so library to static linking. The change can yield double digit percentage improvements to Python scripts but at the cost of larger on-disk space...

http://www.phoronix.com/scan.php?page=news_item&px=Fedora-Static-Link-Python-F32

[Ifr2]

If there is something cheap in this world, that is disk space. I get that a ton of Python programs aren't even 3MB, so this change would more than double their size, but for such a potential improvement of performance is a no brainer to me.

[Ilfirin]

I don't take both reasons. Why you need python faster on modern computer. And also, why you need save 3MB on many GB operating system, on modern computer. But at the end, someone could benefit from some improvement but I can't see one run out of space because of 3MB.

[starshipeleven]

Why you need python faster on modern computer.

If there is something that needs all performance it can get it's stuff using python.
Even fffffing Java is faster

And also, why you need save 3MB on many GB operating system

It's 3MB for a bunch of packages that are currently linking dynamically to python library, but apparently it seems a ton of stuff is linking to python library even if it does not actually use it so the idea is also unlink these applications or python packages.

[CochainComplex]

Speaking from the scientific community. python becomes more and more an opensource alternative to matlab or mathematica. if you have to solve pdes over a decent sized mesh you are thankful for each speed improvement especially if yiu already have some nice gear. sure you could write a c++maybe even fortran code but there you have to spend more time on code maintaining just to make it runable. as a physicist your job is to solve/estimate realworld issues rather developing your new differential forms framework.

[mmrezaie]

It looks like on Ubuntu (19.10) it is already statically linked. This is what I used to do in the clusters to speed up the python script execution time. Usually you describe your pipeline for hpc jobs with python and this would be more than 30% jump in performance time for some loading times (mostly because the libraries are in remote storage).

[HadrienG]

Speaking from the scientific community. python becomes more and more an opensource alternative to matlab or mathematica. if you have to solve pdes over a decent sized mesh you are thankful for each speed improvement especially if yiu already have some nice gear. sure you could write a c++maybe even fortran code but there you have to spend more time on code maintaining just to make it runable. as a physicist your job is to solve/estimate realworld issues rather developing your new differential forms framework.

If you're looking for a researcher-friendly language in use cases where CPU performance matters, you should probably be investigating Julia, not Python.

Many clever minds have expended years of work in trying to make Python usable for CPU-intensive work, and the resulting mixture of highly restricted APIs (to match what can be implemented in C/Fortran + FFI) and "fast python implementations" with incomplete language support, incompatibilities with the cPython ecosystem, and low-performance edge cases can be best described as unsatisfying.

In contrast, Julia has been designed to address the CPU performance problem from the start, and my colleagues who've been trying it out attest that it reasonably lives up to its "decent results from quick prototypes, and tunable to C/Fortran perf levels with a bit of work" promise.

[rene]

3MB? I was surprised how much faster static linked llvm/clang is https://www.youtube.com/watch?v=aZGV_7TfKy8 with results like this maybe it is time to rethink how we do code and shared objects?

[starshipeleven]

maybe it is time to rethink how we do code and shared objects?

Even just infinite and timeless application portability (as you would now be relying on the STABLE kernel-userspace API) would be enough reason for static linking, given that modern disk space constraints (and RAM for that matter) are MUCH different, at least on Desktop.

[ssokolow]

I don't take both reasons. Why you need python faster on modern computer.

The benchmarks game results should speak for themselves.

If you're looking for a researcher-friendly language in use cases where CPU performance matters, you should probably be investigating Julia, not Python.

Many clever minds have expended years of work in trying to make Python usable for CPU-intensive work, and the resulting mixture of highly restricted APIs (to match what can be implemented in C/Fortran + FFI) and "fast python implementations" with incomplete language support, incompatibilities with the cPython ecosystem, and low-performance edge cases can be best described as unsatisfying.

In contrast, Julia has been designed to address the CPU performance problem from the start, and my colleagues who've been trying it out attest that it reasonably lives up to its "decent results from quick prototypes, and tunable to C/Fortran perf levels with a bit of work" promise.

No matter what you're trying to do with a language, "existing ecosystem" tends to be the reason people stick to their existing sub-optimal choices. (eg. I'd love to work exclusively in Rust, but it has no mature Qt bindings or robust, semi-autogenerated ORM migrations or Django ecosystem or mature OpenCV bindings.)