Fedora Stakeholders Debate Statically Linking Python For Better Performance

[coder]

Speaking from the scientific community. python becomes more and more an opensource alternative to matlab or mathematica.

Our data analytics & deep learning folks are using all Python, FWIW. They're pretty rough with C or C++.

Maybe 10 years ago, everybody seemed to be using Matlab or Octave, but numpy did a lot to imbue Python with the core matrix manipulation facilities that set those tools apart. And, as a general programming language, I consider Python far superior to Octave or Matlab.

[CochainComplex]

First, I'll tip my hat to your detailed post. I appreciate the inclusion of references & links.


I didn't say anything about OpenBLAS, and neither had you.


Um, it sounds like a bug in the code, which is mistakenly trying to enable avx512 for all Skylake archs, even though it's only Skylake-SP (or whatever they call the server cores) that has any of it.

But, I guess your point was that you hit an error in some fortran code - thus, you're now aware that it used fortran. Okay, but that point was never in dispute.


This is a silly argument, because I think we both basically agree on the facts. People are still using stuff that was written in Fortran, even if they're not using it for new code.

As for why Intel still cares about it - that's because a handful of venerable libraries use it, which are commonly used - including by some high-profile HPC benchmarks. That doesn't change the fact that probably at least 99.9% of new scientific code is not written in fortran.

I think we both reach the point of agreement.

With BLAS I meant all implementations in general. True, it is not clearly mentiond in my statement. If you analyse the OpenBLAS Package you will find the old BLAS routines as sourcecode If I recall my exploration correctly.

I would also not count fortran as trending language. But in my past of compiling and recompiling a bunch of FOSS scientific tools Octave, OpenFOAM, Elmer, Scilab ...etc I came across a lot of Fortran Code - more than Go or Rust. And btw I m also making use of a decent HPC rig once in a while. Thats why I know that intel (fortan but also c++ etc) compilers are present here and sometimes they outperform gcc builds - but new gcc, gfortran versions (>6) are quite good so I don't have to struggle with compiler compatibility anymore.

And yes Fortran Code is going to be replaced by newer stuff. I did some porting as a Student as well (small project). My Prof wanted to have his Code, written in F77, (Phd stuff from the 80ies) translated to Matlab.

So Fortran is definitely not prospering but it is still embedded in scientifical code and I think it will be sticking around here for some time because its code base is very mature. When it comes to Simulation where you don't have real world testcases numerical stability and accuracy is more than crucial.

[CochainComplex]

Our data analytics & deep learning folks are using all Python, FWIW. They're pretty rough with C or C++.

Maybe 10 years ago, everybody seemed to be using Matlab or Octave, but numpy did a lot to imbue Python with the core matrix manipulation facilities that set those tools apart. And, as a general programming language, I consider Python far superior to Octave or Matlab.

To be honest my favourite language is C++, but if you want to explore some physical Problem in a more "native" - realworld math on paper - like approach is really not the best way to use c++. [OpenFOAM has a nice solution for setting up PDE's with setting files in their (C++)Framework. But this is only one usecase].

Therefore I have used Octave which is a quite good Matlab substitute - but if performance is necessary it is too slow.
Just recently I have tested a "beta" Version package for an octave<->opencl interface to offload some work to the gpu which is really nice btw but my code has to be heavily modified to take advantage of it otherwise the gpu advantages on vector calculations are sacrificed by transfering data CPU to GPU and vice versa.
Octave once implemented JIT but they admit that there has not been much work lately and it is rather picky to compile.

So back to the topic: if I have time I will port the code from octave to python....or maybe finaly to c++ once it is "feature complete".

[coder]

I think we both reach the point of agreement.

*nods*

To be honest my favourite language is C++

Yeah, for big, complex projects - especially where performance or low-level control is critical.

For simple/high-level things that are more involved than I could comfortably do with bash scripts, Python is my go-to. I just wish it were a little more adept at things like string processing & job control (with stream redirection). The latter is where bash really shines, IMO. The former is pretty much the only good argument I recognize for Perl, though I've never embraced it.

That said, I'm open-minded towards (but haven't yet done much experimentation with) options like Rust and Julia.

I have a special fondness for XSLT, but don't get to use it much. It's about as un- C-like as you can get.

[CochainComplex]

btw some comparison done by NASA of different languages used for typical "researcher" operations https://modelingguru.nasa.gov/docs/DOC-2676