Fedora Stakeholders Debate Statically Linking Python For Better Performance

[CochainComplex]

btw some comparison done by NASA of different languages used for typical "researcher" operations https://modelingguru.nasa.gov/docs/DOC-2676

[coder]

I think we both reach the point of agreement.

*nods*

To be honest my favourite language is C++

Yeah, for big, complex projects - especially where performance or low-level control is critical.

For simple/high-level things that are more involved than I could comfortably do with bash scripts, Python is my go-to. I just wish it were a little more adept at things like string processing & job control (with stream redirection). The latter is where bash really shines, IMO. The former is pretty much the only good argument I recognize for Perl, though I've never embraced it.

That said, I'm open-minded towards (but haven't yet done much experimentation with) options like Rust and Julia.

I have a special fondness for XSLT, but don't get to use it much. It's about as un- C-like as you can get.

[CochainComplex]

Our data analytics & deep learning folks are using all Python, FWIW. They're pretty rough with C or C++.

Maybe 10 years ago, everybody seemed to be using Matlab or Octave, but numpy did a lot to imbue Python with the core matrix manipulation facilities that set those tools apart. And, as a general programming language, I consider Python far superior to Octave or Matlab.

To be honest my favourite language is C++, but if you want to explore some physical Problem in a more "native" - realworld math on paper - like approach is really not the best way to use c++. [OpenFOAM has a nice solution for setting up PDE's with setting files in their (C++)Framework. But this is only one usecase].

Therefore I have used Octave which is a quite good Matlab substitute - but if performance is necessary it is too slow.
Just recently I have tested a "beta" Version package for an octave<->opencl interface to offload some work to the gpu which is really nice btw but my code has to be heavily modified to take advantage of it otherwise the gpu advantages on vector calculations are sacrificed by transfering data CPU to GPU and vice versa.
Octave once implemented JIT but they admit that there has not been much work lately and it is rather picky to compile.

So back to the topic: if I have time I will port the code from octave to python....or maybe finaly to c++ once it is "feature complete".

[CochainComplex]

First, I'll tip my hat to your detailed post. I appreciate the inclusion of references & links.


I didn't say anything about OpenBLAS, and neither had you.


Um, it sounds like a bug in the code, which is mistakenly trying to enable avx512 for all Skylake archs, even though it's only Skylake-SP (or whatever they call the server cores) that has any of it.

But, I guess your point was that you hit an error in some fortran code - thus, you're now aware that it used fortran. Okay, but that point was never in dispute.


This is a silly argument, because I think we both basically agree on the facts. People are still using stuff that was written in Fortran, even if they're not using it for new code.

As for why Intel still cares about it - that's because a handful of venerable libraries use it, which are commonly used - including by some high-profile HPC benchmarks. That doesn't change the fact that probably at least 99.9% of new scientific code is not written in fortran.

I think we both reach the point of agreement.

With BLAS I meant all implementations in general. True, it is not clearly mentiond in my statement. If you analyse the OpenBLAS Package you will find the old BLAS routines as sourcecode If I recall my exploration correctly.

I would also not count fortran as trending language. But in my past of compiling and recompiling a bunch of FOSS scientific tools Octave, OpenFOAM, Elmer, Scilab ...etc I came across a lot of Fortran Code - more than Go or Rust. And btw I m also making use of a decent HPC rig once in a while. Thats why I know that intel (fortan but also c++ etc) compilers are present here and sometimes they outperform gcc builds - but new gcc, gfortran versions (>6) are quite good so I don't have to struggle with compiler compatibility anymore.

And yes Fortran Code is going to be replaced by newer stuff. I did some porting as a Student as well (small project). My Prof wanted to have his Code, written in F77, (Phd stuff from the 80ies) translated to Matlab.

So Fortran is definitely not prospering but it is still embedded in scientifical code and I think it will be sticking around here for some time because its code base is very mature. When it comes to Simulation where you don't have real world testcases numerical stability and accuracy is more than crucial.

[coder]

Speaking from the scientific community. python becomes more and more an opensource alternative to matlab or mathematica.

Our data analytics & deep learning folks are using all Python, FWIW. They're pretty rough with C or C++.

Maybe 10 years ago, everybody seemed to be using Matlab or Octave, but numpy did a lot to imbue Python with the core matrix manipulation facilities that set those tools apart. And, as a general programming language, I consider Python far superior to Octave or Matlab.

[coder]

I dont know what distribution you are using but the coder might have a look at this:

Since your not using octave:

https://wiki.octave.org/Octave_and_separate_toolchain

There you will find all the mentioned fortran containing libraries and also acml (lamd) ... according to your name im assuming you will be able to find the dependencies...

MKL can be used to speedup Matlab also Comsol Multiphysics - but proprietary (edit: it became freeware recently). So thats why it is not in your distro, mate.
You might have a look here https://software.intel.com/en-us/one...apack-routines

First, I'll tip my hat to your detailed post. I appreciate the inclusion of references & links.

Try to build OpenBLAS without Fortran. LAPACK is part of OpenBLAS try to exclude the Folder Lapack-netlib ...it will not build.

I didn't say anything about OpenBLAS, and neither had you.

Just recently I have stumbled across an build error if march=skylake is set. it tried to build the fortranparts
with avx512 even if skylake does not imply avx512 per se....

Um, it sounds like a bug in the code, which is mistakenly trying to enable avx512 for all Skylake archs, even though it's only Skylake-SP (or whatever they call the server cores) that has any of it.

But, I guess your point was that you hit an error in some fortran code - thus, you're now aware that it used fortran. Okay, but that point was never in dispute.

So if Fortran is dead why is Intel still selling updated versions of Fortrancompiler ?

If it is used it is simply not dead...

This is a silly argument, because I think we both basically agree on the facts. People are still using stuff that was written in Fortran, even if they're not using it for new code.

As for why Intel still cares about it - that's because a handful of venerable libraries use it, which are commonly used - including by some high-profile HPC benchmarks. That doesn't change the fact that probably at least 99.9% of new scientific code is not written in fortran.

[CochainComplex]

The obvious open source alternative to Matlab is Octave. Is it not good enough? (I've never used either.)

This group has worked on optimizing MatLab Programs. I don't know the current state of their work http://www.sable.mcgill.ca/mclab/projects/

A previous reply mentioned Julia. I imagine that that would be a good alternative if you don't mind recoding.

(The least expensive way I know of for getting a Mathematica license is by buying a Raspberry Pi.)

At the moment I'm using Octave - but in certain situations it is much slower. Octave Devs tried to implement some JIT functionality but there hasn't been a lot of development recently. It also only works with some specific llvm versions.

At my institute I have the possibiltiy to use Mathematica. But at least in a research facility funded by public money FOSS should be prefered over proprietary software.

[CochainComplex]

FFTW does not use it. ACML and MKL aren't even in my distro. Things like LAPACK tend to get replaced with more modern & language-native libraries, like Eigen.

And, as I said in the above post, implementations based on new technology do not use it, such as cuBLAS and MKL-DNN.


You're just citing the long tail of the curve. Until every single one of those stragglers dies off (which they won't, as long as virtually anything still depends on them), people like you will keep claiming it's not dead.

Well, friend, the parrot might not be dead, but he's far from lively.

First I have to admit you are right about FFTW.


I dont know what distribution you are using but the coder might have a look at this:

Since your not using octave:

https://wiki.octave.org/Octave_and_separate_toolchain

There you will find all the mentioned fortran containing libraries and also acml (lamd) ... according to your name im assuming you will be able to find the dependencies...

MKL can be used to speedup Matlab also Comsol Multiphysics - but proprietary (edit: it became freeware recently). So thats why it is not in your distro, mate.
You might have a look here https://software.intel.com/en-us/one...apack-routines

Try to build OpenBLAS without Fortran.
LAPACK is part of OpenBLAS try to exclude the Folder Lapack-netlib ...it will not build.
Just recently I have stumbled across an build error if march=skylake is set. it tried to build the fortranparts
with avx512 even if skylake does not imply avx512 per se....

So if Fortran is dead why is Intel still selling updated versions of Fortrancompiler ?

If it is used it is simply not dead...

[coder]

...but, when that "abstract away the platform so thoroughly that we're not even depending on libc" approach leaks, it leaks hard, resulting in blog posts like this:

https://marcan.st/2017/12/debugging-...o-runtime-bug/

Wow. I'll bet that guy could've saved like a week of debugging, if he'd just run it in valgrind.

Aside from that, it was an impressive effort. Reminds me of a compiler bug I once tracked down, on an embedded platform devoid of such memory checking tools.

[Hugh]

Speaking from the scientific community. python becomes more and more an opensource alternative to matlab or mathematica.

The obvious open source alternative to Matlab is Octave. Is it not good enough? (I've never used either.)

This group has worked on optimizing MatLab Programs. I don't know the current state of their work http://www.sable.mcgill.ca/mclab/projects/

A previous reply mentioned Julia. I imagine that that would be a good alternative if you don't mind recoding.

(The least expensive way I know of for getting a Mathematica license is by buying a Raspberry Pi.)