Clear Linux & Their Love For FMV + dl_platform/dl_hwcap In The Name Of Performance

[jrch2k8]

Anyone else enjoying the fact that Intel has their own C compiler, you know, icc; and yet this super optimized Linux distro they're touting is compiled with gcc?

Besides ICC is closed and quite expensive for an OSS project

[Azultra]

Have Gentoo or Arch users managed to squeeze out so much performance by tweaking their compiler flags too nowadays? About 10 years ago the performance gains one could achieve with Gentoo were small, IIRC less than 10% for most of the tests, nothing like those jaw-dropping Clear Linux numbers.

[r08z]

Have Gentoo or Arch users managed to squeeze out so much performance by tweaking their compiler flags too nowadays? About 10 years ago the performance gains one could achieve with Gentoo were small, IIRC less than 10% for most of the tests, nothing like those jaw-dropping Clear Linux numbers.

Clear Linux does not use -O3 and other exotic flags for everything. They optimize the proper way. They first benchmark, profile, patch when necessary, and LTO, FDO whenever possible. They basically know what the different flags do and the outcome of the choices the make for each package. Most packages are a mixture of either -Os and -O3 switches plus -ffunction-sections -fno-semantic-interposition. On graphics libs they really go head on with -O3 plus -ffat-lto-objects -flto....

[jrch2k8]

Have Gentoo or Arch users managed to squeeze out so much performance by tweaking their compiler flags too nowadays? About 10 years ago the performance gains one could achieve with Gentoo were small, IIRC less than 10% for most of the tests, nothing like those jaw-dropping Clear Linux numbers.

In Arch I do so with Mesa stack but I haven't actually checked performance directly but did notice LTO push the size down by more than half and -fsplit-loops help few loops here and there to get vectorized.(Haswell Xeon E3-1231-v3), so yeah with newer GCCs and hardware flags can make a brutal difference in many fronts(latency, cache coherency, raw speed, size, etc.) depending of what you use.

On Archlinux ARM when recompiling for armv7h and AArch64 I can shove down almost half the total size of the binaries using -Os -ffat-lto-object -flto(on embedded size is pretty important)

Now in general you have to play with the flags to get what is important to you, there is no magic standard flag that will make everything faster

[sdack]

Anyone else enjoying the fact that Intel has their own C compiler, you know, icc; and yet this super optimized Linux distro they're touting is compiled with gcc?

I'm not enjoying it, because I know there are real people behind icc and their efforts count as much as those of anyone else, but I am wondering about it, too. I assume Intel doesn't see much gain in using icc over gcc, but went with it, because it leaves them with the least amount of compatibility issues (meaning, no issues). I've seen them contributing to both GCC and LLVM and they will simply care more for their own hardware than about which software it runs and thus embrace whichever software looks good.

[chithanh]

Using icc to compile your whole distro often is a bad idea. icc is known to take standards-violating shortcuts by default (especially in fp code). There are some Gentoo HPC folks who use icc, but only for their performance-critical compute stuff. All the rest is compiled with gcc or clang.

About 10 years ago the performance gains one could achieve with Gentoo were small, IIRC less than 10% for most of the tests, nothing like those jaw-dropping Clear Linux numbers.

Not sure about 10 years ago, but here is an article from 8 years ago:

http://www.linux-mag.com/id/7574/

It should be noted that right after the introduction of a new architecture, the difference between generic and optimized compiler flags is not that big. But it will grow over time with the introduction of instruction set extensions like SSE or AVX.

[edwaleni]

It's still finicky to install. I install the builds periodically as they roll them out and I get oddities every time. This weeks build gave me screen flashing installation prompts on an Atom-Z. But the same live build on Virtual Box doesn't. Adding the VBox extensions work but show an installation error all the time. The OS still errors out when you install a minimal set to an eMMC drive (like on a SBC). Michael has done a great job at getting the PTS install to work. But I still don't have the OS working properly on my 12c/24t dual Xeon. Trying again this weekend so I can compare PTS results.

Using swupd doesn't bother me for package management, but their installer needs more work. I like to tinker as much as anyone, but I spend too much time messing with it. The futz factor is still too high.

[arjan_intel]

It's still finicky to install. I install the builds periodically as they roll them out and I get oddities every time. This weeks build gave me screen flashing installation prompts on an Atom-Z. But the same live build on Virtual Box doesn't. Adding the VBox extensions work but show an installation error all the time. The OS still errors out when you install a minimal set to an eMMC drive (like on a SBC). Michael has done a great job at getting the PTS install to work. But I still don't have the OS working properly on my 12c/24t dual Xeon. Trying again this weekend so I can compare PTS results.

Using swupd doesn't bother me for package management, but their installer needs more work. I like to tinker as much as anyone, but I spend too much time messing with it. The futz factor is still too high.

obviously I'm a bit sad to read this... I suspect I'm going to need to make some changes to the team and put some more focus on the installer.
if you know of any specific hw or sw things that are broken please let me know

[arjan_intel]

btw if anyone wants to play with CL without installing, you can just do

docker run -it clearlinux/machine-learning

(that gets you a pretty fat docker image with many things already in it)

[edwaleni]

btw if anyone wants to play with CL without installing, you can just do

docker run -it clearlinux/machine-learning

(that gets you a pretty fat docker image with many things already in it)

Excellent idea. I am going to try that.

After some research on why CL won't boot on my 12c/24t dual-Xeon......it was the last board in that generation to not have a full UEFI implementation. CL requires UEFI to run.

My bad, I didn't run the compatibility script ahead of time. Odd, the OS goes through the entire install sequence, locates storage, partitions, reboots, everything until you get a simple blinking cursor in the upper left corner on startup. I was pulling out the Infiniband card, disabling the SAS etc. trying to see what the holdup was.

No worries, we will get CL running full time on another box in the mean time.