Fedora Stakeholders Debate Statically Linking Python For Better Performance

[F.Ultra]

AFAIR this might be for not copy on write modifying the whole executable in memory.

But even on fork() the binary would still call the same address in the library so why would it have to be modified on CoW? (I must be missing something here).

[rene]

But even on fork() the binary would still call the same address in the library so why would it have to be modified on CoW? (I must be missing something here).

instead of having the shared object mapped once in each process using it, you would have a copy-on-write modified version in each process for the relocated jumps all over the place.

[anarki2]

But this is a distribution we are talking about here, their build scripts will rebuild all the statically linked packages when there is an update to the underlying libraries.

Yes. Unless they forget to do that on each and every statically linked package. Or are you implying packages and build scripts are always perfect? That's the whole stupidity of redundancy. It's just one more thing you always need to take care of and remember. Have you heard of database normalization? The whole point of that is the elimination of as much redundancy as possible. Why? Because storing the SAME information in several places is friggin STUPID and can lead to nothing but problems in the future. If you have to do that, you're most definitely doing something wrong.

Because applying the same change to 2 binaries is stupid when you could do that to just one. And it scales to eternity with the number of static links you have. Why on Earth would you do that? Why wouldn't you fix the actual problem instead, that is, improve dynaload performance? The whole world started using dynamic linking because the performance impact was proved to be negligible. How is it that Python still manages to fail in this regard? Surely it's not their fault, and the whole world is wrong instead?

[anarki2]

(emphasis mine)

I was under the impression that Go explicitly encourages statically linked binaries? I could be wrong, but doesn't that entail the exact same issue that you're decrying wrt. the bad old days of MS software development?

Note that I have absolutely no beef with Go.

Ah yes, I was sure someone will imply that if redundancy is bad on Windows, it must be because Windows suxx, therefore Windows developers are idiots. And Linux is awesome, therefore Linux developers are geniuses, therefore, the lessons learned on Windows are somehow irrelevant and Linux apps are somehow immune to the inherent problems of static linking.

If you still can't tell if that was sarcasm or not, let me help you out: GDI and msvcrt.dll were just metaphores and examples of the underlying problem.

Regarding Go, I haven't really looked into it, if they do that, I retract my statement about that.

[Mathias]

Regarding Go, I haven't really looked into it, if they do that, I retract my statement about that.

I think all the go dependencies are statically compiled and only libc is dynamically linked. BTW Rust does the same. Rust can even statically link to two different versions of the same library. Think: Dependency1 depends on fooV1 and Dep2 depends on fooV2, foo will be compiled and linked twice.

[Raka555]

Do you think bundling libssl, gnutls, glibc, curl etc into each application would be wise?

You might as well build it into every binary if the future is to ship each binary with its own set of dynamic libraries via snap/flatpak.

[starshipeleven]

Do you think bundling libssl, gnutls, glibc, curl etc into each application would be wise?

Yeah, it's much more future-proof than relying on "runtimes" or other dependencies even if they are flatpacked.

This is nearly 2020. We need sandboxing by default for all applications anyway, besides in an opensource distro you can get away with your buildbots rebuilding stuff at the tip of a hat.

Decent package managers can already deal with delta updates too, so download bandwith won't change much.

[starshipeleven]

The problem with redundancy is never space, it's rather change management. If there's a vulnerability in libpython, the dudes might patch that but forget about the statically linked stuff in python itself. Or in any other statically linked binary on the system, for that matter. It becomes completely impossible to track. You should never have several copies of the same library. That's why dynamic linking exists in the first place. We've already learned that the hard way, decades ago. Perhaps the most emblematic example is the GDI one:

https://docs.microsoft.com/en-us/sec.../2004/ms04-028

Back in the day, everyone just bundled DLLs instead of proper linking. That's why you had like 500 copies of msvcrt.dll. 99% of them containing vulnerabilities thanks to lazy developers not keeping their stuff up-to-date. Because devs only care about the software "working", and they always forget about the IT part, where it should also stay reliable,secure, and manageable.

And now these clever folks reinvent that (ugly) wheel once again. Instead of fixing dynaload performance issues for their crappy binary. GG. When there's so many so fundamental issues with Python these days, don't be surprised how fast Go takes over.

This is mostly moot in an opensource distro that builds everything from source and has an extensive build system that deals with all packages already. Each time a source file of a lib is changed all packages that depend on it get rebuilt.

OpenSUSE Tumbleweed is using a similar system, each time a library is updated all software relying on it is recompiled and run through the automated QA, once that is cleared, the updated library and the recompiled software are loaded in the repos as updates.

[Raka555]

If there is something cheap in this world, that is disk space. I get that a ton of Python programs aren't even 3MB, so this change would more than double their size, but for such a potential improvement of performance is a no brainer to me.

Disk space use to be very cheap until SSDs came along ...

But a statically linked app loads with sequential reads while dynamic linked app needs to do a lot of seeks. So HDD will have less of a performance hit.

I prefer statically linked apps over packaged flatpak/snaps. You have the same mess with security updates where everything needs to be rebuild/repackaged.

Statically linked apps can actually optimize unused code from the library away (if the library was properly written), while a dynamic lib contains everything from the library in memory. The unused parts latter gets released under memory pressure.

I would very much like to see a distro where everything is statically linked, except for the stuff that really needs dynamic linking.

[starshipeleven]

Yes. Unless they forget to do that on each and every statically linked package. Or are you implying packages and build scripts are always perfect?

How do you think distros work? packages have a dependency manifest.

This won't change for a statically linked distro because the build system still needs this information.
If the dependency manifest is wrong you either have a missing dependency OR useless dependencies

"forgetting to update statically linked packages" is not possible unless the build system is complete garbage and skips some packages at random.