Contemplating A New, Public Linux Daily Kernel Build Server For Ubuntu/Fedora
I've generally relied upon the Ubuntu Mainline Kernel PPA whenever needing a fresh kernel on an Ubuntu Linux system, whether it be that day's Git or using a newer stable/RC release than what's offered in Ubuntu proper. My Ubuntu-based systems doing daily Linux kernel benchmarking tests at LinuxBenchmarking.com have also relied upon it. This package archive has been a great time saver to avoid having to build out a kernel each time while also being of extra value as a nice reference source: any Phoronix readers or anyone else in the community can go ahead and know where to get the kernel packages I used for testing if they wish to try it out for themselves or study their kernel's Kconfig, etc.
Unfortunately, the quality of the Ubuntu Mainline Kernel PPA has degraded in recent time, though in part due to upstream kernel source woes. During the Linux 4.2 kernel cycle, problems plagued a number of systems using the Ubuntu Mainline Kernel packages and even after saying what the problem was in that article, it took a while before it was fixed. Now with the Linux 4.4 kernel, the Ubuntu Mainline Kernel PPA still hasn't been able to build a working kernel for any architecture.
After 4.4-rc1 I mentioned having to workaround Lustre/LNET to build a kernel. We're now almost up to Linux 4.4-rc3 and the Ubuntu repository still doesn't have any daily kernel builds working, so I've still had to keep spinning my own kernels.
Given the woes experienced this year and the slow turnaround time for the problems to be rectified (or at the very least adjusting the Kconfig temporarily to allow working builds), I'm highly considering starting my own daily kernel build server. There would be daily kernel builds from Linux Git with Ubuntu 64-bit packages spun daily and offered online. I'd use that instead for the LinuxBenchmarking.com systems and all of my other Phoronix test systems. This way if there's a problem resulting in the kernel builds to fail, I can quickly fix the problem than having to wait -- weeks -- for the Ubuntu Mainline Kernel PPA to be fixed or the problem worked around upstream.
If going that far and devoting a server to doing daily kernel builds, in maximizing its usefulness, I'd also likely extend it too for daily kernel builds of the Nouveau, Radeon, and Intel -next trees too (the Ubuntu repository has a tracker for DRM-Next and DRM-Intel-Next, but not others) if wanting to easily test the experimental graphics driver code. Beyond that, in the past I did experiment with some Phoronix Test Suite modules for picking up patches automatically off mailing lists to then patch a code-base and run some benchmarks. I could see about adapting that too for spinning extra daily kernels with certain out-of-tree patches applied, such as has been of interest lately with AMDGPU PowerPlay and Nouveau re-clocking patches, etc. Lots of this automated daily kernel building could be triggered and powered by Phoromatic that already automates all of our daily Linux benchmarking systems.
Another avenue to consider would also be doing identical kernel builds also for Fedora. While many tests at Phoronix, LinuxBenchmarking.com, etc, are done under Ubuntu, Fedora Linux is the second most used distribution here for running benchmarks and I have dozens of Fedora systems around. So I'd also look to be doing identical kernel builds for Fedora.
One other item tieing into this is also PTS Desktop Live, the live Linux distribution I was working on in 2009~2010 that provided an automated, standardized Linux platform from which benchmarks could be launched and would provide a standard platform for ensuring the results across hardware are comparable by keeping the software components the same. I've been meaning to re-work on this Linux benchmarking distribution, but alas am continually short on time. My latest plans for its re-launch are to re-base it against Fedora and to be issuing new builds of it bi-weekly or monthly that would include the latest kernel, Mesa, LLVM, GCC, etc, at the time. So on the same version/build all the software would still be maintained to ensure reproducible/comparable results but also for making an easy live environment to quickly smoke-test new drivers on different hardware, etc. Hopefully in 2016 I'll get back to working on that and organize a release after I finish developing the new version of OpenBenchmarking.org.
Anyhow, those are some of the extensions I'm thinking about right now. Any feedback is welcome and much appreciated by contacting me directly (@MichaelLarabel on Twitter or MichaelLarabel.com) or by commenting on this article in our forums.