Clang LTO Support Looks Like It Could Land For Linux 5.12

[CochainComplex]

I saw one to three objtool errors but they weren't fatal for my build. Have you tried to disable Retpolines? I use a minimal config for my system and disable most stuff I don't need. I use -O3 -march=native and aggressive compiler flags (polly) and could get it working on my machine (Haswell-EP + Vega 56) with LLVM/Clang 11.

for march=native I also have to get rid of retpoline otherwise build will fail with a huge amout of objtool errors,

[AdelKS]

I saw one to three objtool errors but they weren't fatal for my build. Have you tried to disable Retpolines? I use a minimal config for my system and disable most stuff I don't need. I use -O3 -march=native and aggressive compiler flags (polly) and could get it working on my machine (Haswell-EP + Vega 56) with LLVM/Clang 11.

ms178 I tried with modrobed-db but to no avail, linking takes ages then fails with thousands of objtool errors, then tries again then fails in an infinite loop. I will try to disable retpolines to see. Also, please give me a reference for your aggressive compiler flags and how to enable them for the kernel, I am interested.

[aspen]

The main issue with building the kernel with Clang is that DKMS fucks up, at least on Arch. Rather sad, because it's way too annoying to manually configure DKMS build scripts to actually use something that isn't gcc.

[xpris]

The main issue with building the kernel with Clang is that DKMS fucks up, at least on Arch. Rather sad, because it's way too annoying to manually configure DKMS build scripts to actually use something that isn't gcc.

OpenMandriva ships Clang kernel as option in past release and Clang kernel should be default one in upcoming. So not that hard.

[ms178]

ms178 I tried with modrobed-db but to no avail, linking takes ages then fails with thousands of objtool errors, then tries again then fails in an infinite loop. I will try to disable retpolines to see. Also, please give me a reference for your aggressive compiler flags and how to enable them for the kernel, I am interested.

I use the following for KCFLAGS / KCPPFLAGS: "-O3 -march=native -mtune=native -mllvm -polly -mllvm -polly-parallel -fopenmp -mllvm -polly-vectorizer=stripmine -mllvm -polly-omp-backend=LLVM -mllvm -polly-num-threads=24 -mllvm -polly-scheduling=dynamic -mllvm -polly-scheduling-chunksize=1 -fno-math-errno -fno-trapping-math -feliminate-unused-debug-types -fno-semantic-interposition -falign-functions=32"

KLDFLAGS="-Wl,-O2 -Wl,--as-needed"

There are several ways how to set these, either by modifying the top level makefile (search for HOSTCFLAGS) or by adding these to your make command line. You have to adjust the polly-num-threads option to the max threads your CPU has.

Concerning the errors you have seen, I mentioned two Kernel options which are likely the culprit, you need to disable these settings in your Kernel configuration, Retpolines is hidden behind the "avoid speculation in the kernel" option which you would need to disable and make also sure to not trim the generated symbols which would lead to modpost errors.

[AdelKS]

Thanks! ms178

[reba]

After reading the article I was tempted to try out my own Clang+LLVM compiled kernel (from xanmod) and... what can I say, it compiled beautifully and I'm currently running it

Even the VirtualBox module could be compiled and the VMs are running without a problem! O_O

If you're interested here's the script I came up with:

#!/bin/bash

echo 'Installing the build environment...'
echo 'See: https://wiki.debian.org/BuildADebianKernelPackage "Install the Required Packages"'
# sudo apt-get install build-essential linux-source bc kmod cpio flex libncurses5-dev libelf-dev libssl-dev
sudo apt-get install build-essential bison bc kmod cpio flex libncurses5-dev libelf-dev libssl-dev

echo 'Installing Clang...'
echo 'See: https://www.kernel.org/doc/html/v5.7/kbuild/llvm.html'
sudo apt-get install clang

echo 'Installing LLVM and LLD...'
sudo apt-get install llvm lld

# echo 'Extracting the kernel source to the current directory and changing into...'
# echo 'See: https://wiki.debian.org/BuildADebianKernelPackage "Extracting the Kernel Source"'
# tar xavf /usr/src/linux-source-*.tar.xz
# cd linux-source-*

echo 'Cloning xanmod kernel...'
git clone https://github.com/xanmod/linux.git
cd linux

echo 'Setting environment for "make" to use Clang, LLVM, march, etc...'
echo 'See: https://unix.stackexchange.com/a/545385'
export CC="clang"
export HOSTCC="clang"
export LLVM="1"
export KCFLAGS="-O2 -march=native -pipe"
export KCPPFLAGS="-O2 -march=native -pipe"

# First run only, otherwise this would overwrite custom settings
# echo 'Configuring the kernel...'
# echo 'See: https://wiki.debian.org/BuildADebianKernelPackage "Configuring the Kernel"'
# make defconfig

echo 'Setting new "localversion"...'
echo "-xanmod-llvm-native" > localversion

echo 'Building the kernel to a deb-package...'
echo 'See: https://wiki.debian.org/BuildADebianKernelPackage "Building the Debian Package"'
time nice make -j `nproc` bindeb-pkg

Then just install the packages:

sudo dpkg -i ../linux-*.deb

After rebooting the kernel can be selected from GRUB.

Virtualbox will not work as the vboxdrv-module isn't loaded and has to be compiled manually.
I reinstalled the whole virtualbox-dkms package but it fails when compiling for the active kernel:

sudo apt-get --reinstall install virtualbox-dkms

...
Unpacking virtualbox-dkms (6.1.22-dfsg-2) over (6.1.22-dfsg-2) ...
Setting up virtualbox-dkms (6.1.22-dfsg-2) ...
Loading new virtualbox-6.1.22 DKMS files...
Building for 5.13.1-xanmod-llvm-native
Building initial module for 5.13.1-xanmod-llvm-native
Error! Bad return status for module build on kernel: 5.13.1-xanmod-llvm-native (x86_64)
Consult /var/lib/dkms/virtualbox/6.1.22/build/make.log for more information.
dpkg: error processing package virtualbox-dkms (--configure):
installed virtualbox-dkms package post-installation script subprocess returned error exit status 10
Errors were encountered while processing:
virtualbox-dkms
E: Sub-process /usr/bin/dpkg returned an error code (1)

So I thought: "Hm. Maybe the CC (which is usually gcc) for compiling the module does not like the Clang compiled kernel. So, what about compiling the module with Clang, too?"

So I edited the Makefile of the vboxdrv to add the same compiler-options as for the kernel:

nano /usr/src/linux-headers-5.13.1-xanmod-llvm-native/Makefile

Just added these line to the top of the file:

CC=clang
HOSTCC=clang
LLVM=1

and the recompilation of the module worked:

sudo dpkg --configure virtualbox-dkms

...
Loading new virtualbox-6.1.22 DKMS files...
Building for 5.13.1-xanmod-llvm-native
Building initial module for 5.13.1-xanmod-llvm-native
Done.

vboxdrv.ko:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/5.13.1-xanmod-llvm-native/updates/dkms/

vboxnetadp.ko:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/5.13.1-xanmod-llvm-native/updates/dkms/

vboxnetflt.ko:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/5.13.1-xanmod-llvm-native/updates/dkms/

depmod...

DKMS: install completed.

Trying

modprobe vboxdrv

loaded the module without complains:

$ lsmod | grep vbox
vboxnetadp 28672 0
vboxnetflt 32768 0
vboxdrv 438272 2 vboxnetadp,vboxnetflt

I am totally astounded how easy this was, even more so coming for a noob that I am, wowzers!

Big kudos and thumbs up from me to everyone who contributed to make this whole thing absolutely seamless!

$ uname -a
Linux debian 5.13.1-xanmod-llvm-native #1 SMP Sat Jul 10 12:23:59 CEST 2021 x86_64 GNU/Linux

[AdelKS]

reba I think you missed one more option to give to `make`, which is `LLVM_IAS=1`. If you don't use it, something is not enabled. I don't know exactly what.

Edit: the patch says to do the following, to enable Thin LTO for example. I think you didn't enable LTO at all.

$ make LLVM=1 LLVM_IAS=1 defconfig
$ scripts/config -e LTO_CLANG_THIN
$ make LLVM=1 LLVM_IAS=1

And when LTO is enabled, the compilation will most probably fail with objtool errors, you can't compile a "normal" config file for now.

Edit 2: You haven't said at all that that's how you enable LTO haha, I misread, my bad.

[reba]

reba I think you missed one more option to give to `make`, which is `LLVM_IAS=1`. If you don't use it, something is not enabled. I don't know exactly what.

Yes, I've seen it but I don't know what this "integrated assembler" is supposed to mean?

Edit: the patch says to do the following, to enable Thin LTO for example. I think you didn't enable LTO at all.

$ make LLVM=1 LLVM_IAS=1 defconfig
$ scripts/config -e LTO_CLANG_THIN
$ make LLVM=1 LLVM_IAS=1

And when LTO is enabled, the compilation will most probably fail with objtool errors, you can't compile a "normal" config file for now.

Edit 2: You haven't said at all that that's how you enable LTO haha, I misread, my bad.

No, LTO and GPO are not enabled AFAIK in my setup. That would be step two

Their documentation mentions some parameters regarding LTO:

-flto, -fno-lto
Enable LTO in ‘full’ mode
-flto-jobs=<arg>
Controls the backend parallelism of -flto=thin (default of 0 means the number of threads will be derived from the number of CPUs detected)
-flto=<arg>
Set LTO mode to either ‘full’ or ‘thin’

As I said, that's step two - I was just awed Clang and LLVM work AT ALL
Kernel compilation sadly takes some real time and there's only that much in a day.


Edit:
At least for this 5.10.7

grep -R -i "LTO_CLANG" linux

did not return any hits.

[leandrolnh]

Edit:
At least for this 5.10.7

grep -R -i "LTO_CLANG" linux

did not return any hits.

That's the discussion of the news: LTO with Clang is not in mainstream yet. I don't think xanmod has added support for it either.
That's why you haven't found it.