Fedora Stakeholders Debate Statically Linking Python For Better Performance

[Britoid]

Even just infinite and timeless application portability (as you would now be relying on the STABLE kernel-userspace API) would be enough reason for static linking, given that modern disk space constraints (and RAM for that matter) are MUCH different, at least on Desktop.

Do you think bundling libssl, gnutls, glibc, curl etc into each application would be wise?

[rene]

Even just infinite and timeless application portability (as you would now be relying on the STABLE kernel-userspace API) would be enough reason for static linking, given that modern disk space constraints (and RAM for that matter) are MUCH different, at least on Desktop.

I did not say to use static linking. I actually don't support that very much as each library security update requires rebuilding or relinking all the executables. I mean to think about something more innovative.

[CochainComplex]

If you're looking for a researcher-friendly language in use cases where CPU performance matters, you should probably be investigating Julia, not Python.

Many clever minds have expended years of work in trying to make Python usable for CPU-intensive work, and the resulting mixture of highly restricted APIs (to match what can be implemented in C/Fortran + FFI) and "fast python implementations" with incomplete language support, incompatibilities with the cPython ecosystem, and low-performance edge cases can be best described as unsatisfying.

In contrast, Julia has been designed to address the CPU performance problem from the start, and my colleagues who've been trying it out attest that it reasonably lives up to its "decent results from quick prototypes, and tunable to C/Fortran perf levels with a bit of work" promise.

Thx for your suggestion.

[discordian]

I did not say to use static linking. I actually don't support that very much as each library security update requires rebuilding or relinking all the executables. I mean to think about something more innovative.

Theres really no third option, either you put the code in the app or you use it from an external file (shared lib)?

If you talk about making shared libs faster, then yes there is actually alot that could be done when it comes to ELF libraries. If you call a public function foo from a function bar, from the same library, then the library has to go through a stub instead of calling the code that could be a small jump away.
This is necessary if you want to dynamically replace the function bar, but quite often you do not require or want that. Outside of libc (replacing/instrumenting malloc, fakeroot) this should be disabled IMHO.

Just checked on debian, python3 is linked statically here.

[rene]

Theres really no third option, either you put the code in the app or you use it from an external file (shared lib)?

If you talk about making shared libs faster, then yes there is actually alot that could be done when it comes to ELF libraries. If you call a public function foo from a function bar, from the same library, then the library has to go through a stub instead of calling the code that could be a small jump away.
This is necessary if you want to dynamically replace the function bar, but quite often you do not require or want that. Outside of libc (replacing/instrumenting malloc, fakeroot) this should be disabled IMHO.

Just checked on debian, python3 is linked statically here.

you write "there is no third option" and then you continue discussing a third option of making ELF libraries faster ;-) Another option would be to use byte code and JIT it, which might result in huge speedups for either small programs not using 99% of the loaded libraries, or SIMD stuff where this could yield more optimized code than currently static compiled code not optimized for the latest and greatest AVX during generic pre-compile time.

[anarki2]

The problem with redundancy is never space, it's rather change management. If there's a vulnerability in libpython, the dudes might patch that but forget about the statically linked stuff in python itself. Or in any other statically linked binary on the system, for that matter. It becomes completely impossible to track. You should never have several copies of the same library. That's why dynamic linking exists in the first place. We've already learned that the hard way, decades ago. Perhaps the most emblematic example is the GDI one:

https://docs.microsoft.com/en-us/sec.../2004/ms04-028

Back in the day, everyone just bundled DLLs instead of proper linking. That's why you had like 500 copies of msvcrt.dll. 99% of them containing vulnerabilities thanks to lazy developers not keeping their stuff up-to-date. Because devs only care about the software "working", and they always forget about the IT part, where it should also stay reliable,secure, and manageable.

And now these clever folks reinvent that (ugly) wheel once again. Instead of fixing dynaload performance issues for their crappy binary. GG. When there's so many so fundamental issues with Python these days, don't be surprised how fast Go takes over.

[F.Ultra]

you write "there is no third option" and then you continue discussing a third option of making ELF libraries faster ;-) Another option would be to use byte code and JIT it, which might result in huge speedups for either small programs not using 99% of the loaded libraries, or SIMD stuff where this could yield more optimized code than currently static compiled code not optimized for the latest and greatest AVX during generic pre-compile time.

Shouldn't be needed to go the full byte code route, the compiler could output a jmp $0000000000000000 and record the place and destionation in a table instead of doing it at the stub-level and then overwrite all the locations upon load. I think people went with the stub option back in the day to not have to go over all the file so that load times where faster?!

[F.Ultra]

The problem with redundancy is never space, it's rather change management. If there's a vulnerability in libpython, the dudes might patch that but forget about the statically linked stuff in python itself. Or in any other statically linked binary on the system, for that matter. It becomes completely impossible to track. You should never have several copies of the same library. That's why dynamic linking exists in the first place. We've already learned that the hard way, decades ago. Perhaps the most emblematic example is the GDI one:

https://docs.microsoft.com/en-us/sec.../2004/ms04-028

Back in the day, everyone just bundled DLLs instead of proper linking. That's why you had like 500 copies of msvcrt.dll. 99% of them containing vulnerabilities thanks to lazy developers not keeping their stuff up-to-date. Because devs only care about the software "working", and they always forget about the IT part, where it should also stay reliable,secure, and manageable.

And now these clever folks reinvent that (ugly) wheel once again. Instead of fixing dynaload performance issues for their crappy binary. GG. When there's so many so fundamental issues with Python these days, don't be surprised how fast Go takes over.

But this is a distribution we are talking about here, their build scripts will rebuild all the statically linked packages when there is an update to the underlying libraries.

[rene]

Shouldn't be needed to go the full byte code route, the compiler could output a jmp $0000000000000000 and record the place and destionation in a table instead of doing it at the stub-level and then overwrite all the locations upon load. I think people went with the stub option back in the day to not have to go over all the file so that load times where faster?!

AFAIR this might be for not copy on write modifying the whole executable in memory.

[ermo]

The problem with redundancy is never space, it's rather change management. If there's a vulnerability in libpython, the dudes might patch that but forget about the statically linked stuff in python itself. Or in any other statically linked binary on the system, for that matter. It becomes completely impossible to track. You should never have several copies of the same library. That's why dynamic linking exists in the first place. We've already learned that the hard way, decades ago. Perhaps the most emblematic example is the GDI one:

https://docs.microsoft.com/en-us/sec.../2004/ms04-028

Back in the day, everyone just bundled DLLs instead of proper linking. That's why you had like 500 copies of msvcrt.dll. 99% of them containing vulnerabilities thanks to lazy developers not keeping their stuff up-to-date. Because devs only care about the software "working", and they always forget about the IT part, where it should also stay reliable,secure, and manageable.

And now these clever folks reinvent that (ugly) wheel once again. Instead of fixing dynaload performance issues for their crappy binary. GG. When there's so many so fundamental issues with Python these days, don't be surprised how fast Go takes over.

(emphasis mine)

I was under the impression that Go explicitly encourages statically linked binaries? I could be wrong, but doesn't that entail the exact same issue that you're decrying wrt. the bad old days of MS software development?

Note that I have absolutely no beef with Go.