Clang LTO Support Looks Like It Could Land For Linux 5.12

[DanglingPointer]

dkms developers need to fix this!

dkms needs to be able to dynamically figure out what build tools were used to build the kernel and use the same ones for the modules! Other's have had the same issue and raised a ticket against dkms. dkms was invented by Dell and github repo and issue tracker ticket for the bug is here...

https://github.com/dell/dkms/issues/124

If you are having the same problems with modules or like me with Virtualbox using dkms with a host compiled by llvm-clang; please add your voice and comment in the ticket above! Sooner this is fixed the sooner we can all benefit from a more optimised linux kernel for hosts running VMs.

[DanglingPointer]

Hi there. I just re-played the stuff in my posting and made some adjustments.
For example bison was missing as a dependency and I added some snippets and explainations.
And, yes, you're supposed to edit the Makefile of the linux headers.

The module zramswap also breaks *sigh* so there might be others as well that are in need for a recompile.
Maybe, after all, I am doing something wrong?

Jul 10 13:26:33 debian root[12072]: Starting Zram
Jul 10 13:26:33 debian zramswap[12072]: <13>Jul 10 13:26:33 root: Starting Zram
Jul 10 13:26:33 debian zramswap[12074]: modprobe: FATAL: Module zram not found in directory /lib/modules/5.13.1-xanmod-llvm-native
Jul 10 13:26:33 debian root[12075]: Error: inserting the zram kernel module
Jul 10 13:26:33 debian zramswap[12075]: <13>Jul 10 13:26:33 root: Error: inserting the zram kernel module
Jul 10 13:26:33 debian systemd[1]: zramswap.service: Main process exited, code=exited, status=1/FAILURE

So did your zram fail because dkms used gcc to try and compile it instead of "cc" aliasing clang?

my vbox module is getting explicitly compiled by dkms with gcc and failing. switching cc to clang didn't do anything.

I think this is a dkms issue. DKMS needs to use "cc" instead of explicit compiler calls to /usr/bin/gcc and it suite of tools

No one is going to use llvm clang lto'ed kernels if this isn't sorted (other than those using openMandriva).

[reba]

Thanks in advance!

Hi there. I just re-played the stuff in my posting and made some adjustments.
For example bison was missing as a dependency and I added some snippets and explainations.
And, yes, you're supposed to edit the Makefile of the linux headers.

The module zramswap also breaks *sigh* so there might be others as well that are in need for a recompile.
Maybe, after all, I am doing something wrong?

Jul 10 13:26:33 debian root[12072]: Starting Zram
Jul 10 13:26:33 debian zramswap[12072]: <13>Jul 10 13:26:33 root: Starting Zram
Jul 10 13:26:33 debian zramswap[12074]: modprobe: FATAL: Module zram not found in directory /lib/modules/5.13.1-xanmod-llvm-native
Jul 10 13:26:33 debian root[12075]: Error: inserting the zram kernel module
Jul 10 13:26:33 debian zramswap[12075]: <13>Jul 10 13:26:33 root: Error: inserting the zram kernel module
Jul 10 13:26:33 debian systemd[1]: zramswap.service: Main process exited, code=exited, status=1/FAILURE

[DanglingPointer]

Virtualbox will not work as the vboxdrv-module isn't loaded and has to be compiled manually.
I made aptitude reinstall the virtualbox-dkms package but it fails when compiling for the active kernel.

So I thought "Hm. Maybe the CC (which is usually gcc) for compiling the module does not like the Clang compiled kernel. So, what about compiling the module with Clang, too?"

So I edited the Makefile of the vboxdrv to add the same compiler-options as for the kernel:

nano /usr/src/linux-headers-5.10.7-xanmod-llvm-native

Just added these on top:

CC=clang
HOSTCC=clang
LLVM=1

and the recompilation of the module worked!

Trying

modprobe vboxdrv

loaded the module without complains.

Hi reba
I have a similar problem where my vbox v6.1.22 fails to run any guests and its because the vboxdrv fails to compile through dkms.
I naturally thought it was something for Virtualbox to fix so I filed a bug there... https://www.virtualbox.org/ticket/20425

Not sure really who needs to fix this but it needs to get fixed, how are kernel modules going to get compile dynamically when changed if the Linux kernel is built with clang and lto=thin?

How did you fix it? You mentioned you edited the Makefile of the vboxdrv but your quote above does a "nano" of the linux-header's Makefile? I'm a little confused and would appreciate if you could elaborate as I too would like to use clang lto'ed kernel. I've rolled back to gcc built kernel as I can't work without virtualbox.

Thanks in advance!

[reba]

That's the discussion of the news: LTO with Clang is not in mainstream yet. I don't think xanmod has added support for it either.
That's why you haven't found it.

Yes. I thought there were in but not yet enabled. Looks like they're just about to get in. It somehow reads they're just about to be enabled for 5.12.
I don't think xanmod would do this amount of work downstream but rely on upstream.

Anyways, my bod^M script is ready.

[aspen]

OpenMandriva ships Clang kernel as option in past release and Clang kernel should be default one in upcoming. So not that hard.

Huh, you got a link to code that shows how they do it? I'm interested.

[leandrolnh]

Edit:
At least for this 5.10.7

grep -R -i "LTO_CLANG" linux

did not return any hits.

That's the discussion of the news: LTO with Clang is not in mainstream yet. I don't think xanmod has added support for it either.
That's why you haven't found it.

[reba]

reba I think you missed one more option to give to `make`, which is `LLVM_IAS=1`. If you don't use it, something is not enabled. I don't know exactly what.

Yes, I've seen it but I don't know what this "integrated assembler" is supposed to mean?

Edit: the patch says to do the following, to enable Thin LTO for example. I think you didn't enable LTO at all.

$ make LLVM=1 LLVM_IAS=1 defconfig
$ scripts/config -e LTO_CLANG_THIN
$ make LLVM=1 LLVM_IAS=1

And when LTO is enabled, the compilation will most probably fail with objtool errors, you can't compile a "normal" config file for now.

Edit 2: You haven't said at all that that's how you enable LTO haha, I misread, my bad.

No, LTO and GPO are not enabled AFAIK in my setup. That would be step two

Their documentation mentions some parameters regarding LTO:

-flto, -fno-lto
Enable LTO in ‘full’ mode
-flto-jobs=<arg>
Controls the backend parallelism of -flto=thin (default of 0 means the number of threads will be derived from the number of CPUs detected)
-flto=<arg>
Set LTO mode to either ‘full’ or ‘thin’

As I said, that's step two - I was just awed Clang and LLVM work AT ALL
Kernel compilation sadly takes some real time and there's only that much in a day.


Edit:
At least for this 5.10.7

grep -R -i "LTO_CLANG" linux

did not return any hits.

[AdelKS]

reba I think you missed one more option to give to `make`, which is `LLVM_IAS=1`. If you don't use it, something is not enabled. I don't know exactly what.

Edit: the patch says to do the following, to enable Thin LTO for example. I think you didn't enable LTO at all.

$ make LLVM=1 LLVM_IAS=1 defconfig
$ scripts/config -e LTO_CLANG_THIN
$ make LLVM=1 LLVM_IAS=1

And when LTO is enabled, the compilation will most probably fail with objtool errors, you can't compile a "normal" config file for now.

Edit 2: You haven't said at all that that's how you enable LTO haha, I misread, my bad.

[reba]

After reading the article I was tempted to try out my own Clang+LLVM compiled kernel (from xanmod) and... what can I say, it compiled beautifully and I'm currently running it

Even the VirtualBox module could be compiled and the VMs are running without a problem! O_O

If you're interested here's the script I came up with:

#!/bin/bash

echo 'Installing the build environment...'
echo 'See: https://wiki.debian.org/BuildADebianKernelPackage "Install the Required Packages"'
# sudo apt-get install build-essential linux-source bc kmod cpio flex libncurses5-dev libelf-dev libssl-dev
sudo apt-get install build-essential bison bc kmod cpio flex libncurses5-dev libelf-dev libssl-dev

echo 'Installing Clang...'
echo 'See: https://www.kernel.org/doc/html/v5.7/kbuild/llvm.html'
sudo apt-get install clang

echo 'Installing LLVM and LLD...'
sudo apt-get install llvm lld

# echo 'Extracting the kernel source to the current directory and changing into...'
# echo 'See: https://wiki.debian.org/BuildADebianKernelPackage "Extracting the Kernel Source"'
# tar xavf /usr/src/linux-source-*.tar.xz
# cd linux-source-*

echo 'Cloning xanmod kernel...'
git clone https://github.com/xanmod/linux.git
cd linux

echo 'Setting environment for "make" to use Clang, LLVM, march, etc...'
echo 'See: https://unix.stackexchange.com/a/545385'
export CC="clang"
export HOSTCC="clang"
export LLVM="1"
export KCFLAGS="-O2 -march=native -pipe"
export KCPPFLAGS="-O2 -march=native -pipe"

# First run only, otherwise this would overwrite custom settings
# echo 'Configuring the kernel...'
# echo 'See: https://wiki.debian.org/BuildADebianKernelPackage "Configuring the Kernel"'
# make defconfig

echo 'Setting new "localversion"...'
echo "-xanmod-llvm-native" > localversion

echo 'Building the kernel to a deb-package...'
echo 'See: https://wiki.debian.org/BuildADebianKernelPackage "Building the Debian Package"'
time nice make -j `nproc` bindeb-pkg

Then just install the packages:

sudo dpkg -i ../linux-*.deb

After rebooting the kernel can be selected from GRUB.

Virtualbox will not work as the vboxdrv-module isn't loaded and has to be compiled manually.
I reinstalled the whole virtualbox-dkms package but it fails when compiling for the active kernel:

sudo apt-get --reinstall install virtualbox-dkms

...
Unpacking virtualbox-dkms (6.1.22-dfsg-2) over (6.1.22-dfsg-2) ...
Setting up virtualbox-dkms (6.1.22-dfsg-2) ...
Loading new virtualbox-6.1.22 DKMS files...
Building for 5.13.1-xanmod-llvm-native
Building initial module for 5.13.1-xanmod-llvm-native
Error! Bad return status for module build on kernel: 5.13.1-xanmod-llvm-native (x86_64)
Consult /var/lib/dkms/virtualbox/6.1.22/build/make.log for more information.
dpkg: error processing package virtualbox-dkms (--configure):
installed virtualbox-dkms package post-installation script subprocess returned error exit status 10
Errors were encountered while processing:
virtualbox-dkms
E: Sub-process /usr/bin/dpkg returned an error code (1)

So I thought: "Hm. Maybe the CC (which is usually gcc) for compiling the module does not like the Clang compiled kernel. So, what about compiling the module with Clang, too?"

So I edited the Makefile of the vboxdrv to add the same compiler-options as for the kernel:

nano /usr/src/linux-headers-5.13.1-xanmod-llvm-native/Makefile

Just added these line to the top of the file:

CC=clang
HOSTCC=clang
LLVM=1

and the recompilation of the module worked:

sudo dpkg --configure virtualbox-dkms

...
Loading new virtualbox-6.1.22 DKMS files...
Building for 5.13.1-xanmod-llvm-native
Building initial module for 5.13.1-xanmod-llvm-native
Done.

vboxdrv.ko:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/5.13.1-xanmod-llvm-native/updates/dkms/

vboxnetadp.ko:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/5.13.1-xanmod-llvm-native/updates/dkms/

vboxnetflt.ko:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/5.13.1-xanmod-llvm-native/updates/dkms/

depmod...

DKMS: install completed.

Trying

modprobe vboxdrv

loaded the module without complains:

$ lsmod | grep vbox
vboxnetadp 28672 0
vboxnetflt 32768 0
vboxdrv 438272 2 vboxnetadp,vboxnetflt

I am totally astounded how easy this was, even more so coming for a noob that I am, wowzers!

Big kudos and thumbs up from me to everyone who contributed to make this whole thing absolutely seamless!

$ uname -a
Linux debian 5.13.1-xanmod-llvm-native #1 SMP Sat Jul 10 12:23:59 CEST 2021 x86_64 GNU/Linux