Fedora Stakeholders Debate Statically Linking Python For Better Performance

[Vistaus]

If you're looking for a researcher-friendly language in use cases where CPU performance matters, you should probably be investigating Julia, not Python.

I thought Fortran was the choice of many scientists???

[andyprough]

If they want better performance - maybe they should just use something that is not Fedora??

[HadrienG]

I thought Fortran was the choice of many scientists???

It has long been, and remains a timeless classic in areas where maximal performance is required like HPC. But these days, in most areas of scientific software, Fortran tends to be displaced by higher-level languages like Matlab or more recently Python and Julia.

[F.Ultra]

instead of having the shared object mapped once in each process using it, you would have a copy-on-write modified version in each process for the relocated jumps all over the place.

Why? The library will still jump to the same addresses regardless of how many processes that link to it. I don't see how the stubs solves anything other than making the initial load of a binary faster.

[oleid]

If there is something cheap in this world, that is disk space. I get that a ton of Python programs aren't even 3MB, so this change would more than double their size, but for such a potential improvement of performance is a no brainer to me.

This is not about statically linking libpython to python scripts, it's about statically linking libpython to the python executable - the interpreter itself. The scripts don't change _at all_.

There are other programs, which use libpython, i.e. some $WINDOW_MANAGER, which uses it for internal scripting stuff. They will possibly get statically linked, too.

[pal666]

https://www.akkadia.org/drepper/no_static_linking.html

[pal666]

s/depending of workload/in synthetic tests/
btw, you can make python much faster by avoiding python

[endrebjorsvik]

I think all the go dependencies are statically compiled and only libc is dynamically linked. BTW Rust does the same. Rust can even statically link to two different versions of the same library. Think: Dependency1 depends on fooV1 and Dep2 depends on fooV2, foo will be compiled and linked twice.

Idiomatic Go does not even depend on libc. Go encourages you to not use libc, and linking against it is disabled by default when you cross-compile.
You are correct regarding how other dependencies work in Go. All dependencies are statically linked into one fat binary together with the supplied Go-runtime. Another thing to note about dependencies in Go projects, is that they very often end up picking a plethora of random libraries from east and west on Github. As an example, consider the dependency list for Kubernetes. This is also partly encouraged by the Go eco-system and toolchain.

[pal666]

Why? The library will still jump to the same addresses regardless of how many processes that link to it.

it doesn't work that way. for it someone has to globally decide which part of address space every library on machine takes, otherwise you'll have two libraries or library and executable clash. prelink tried to do that and prelink is obsolete. and that was even before address space layout randomization

[pal666]

rebuilding/relinking isn't a huge issue with ccache

unless you are using lto/pgo like subj