The Fedora Linux Rawhide Kernel Is Becoming Too Slow With Its Many Debug Options

Fedora is one of the Linux distributions that ships with a plethora of debug options during its "Rawhide" development phase to ease in diagnosing issues that turn up during testing rather than building everything in a release mode during the development cycle. While these debug options are good for debugging, the performance impact continues adding up and reaching a point that the Fedora Rawhide debug kernel is too slow for some tasks.

Richard Jones of Red Hat recently was looking at the debug options used in the Fedora Rawhide debug kernels due to becoming too slow. The longtime Linux developer and engineer from Red Hat's Virtualization Group explained:

"The current Fedora Rawhide kernels are too slow to run libguestfs tests when doing Koji builds. These run in a qemu VM, running the Rawhide kernel, emulated using software virtualization (ie. TCG). They now time out because these kernels are so slow. Until fairly recently they were slow but working."

Thus he jumped down the rabbit hole looking at the performance cost to various debug options used by the Rawhide kernel.

The Fedora Rawhide kernel with all its debug options was found to a 143% increase in the run-time of his test case, going from 12.3 seconds without the debug options to over 30 seconds.

The most expensive debug options used by the Fedora Rawhide Linux kernel.

Among the most expensive debug options used by the Fedora Rawhide kernel right now are PROVE_LOCKING, LOCK_STAT, DEBUG_WW_MUTEX_SLOWPATH, DEBUG_KMEMLEAK, DEBUG_LOCK_ALLOC, and PAGE_TABLE_CHECK_ENFORCED.

On the Fedora kernel mailing list is now this thread outlining his results. Hopefully this will lead to a broader discussion over the Fedora Rawhide kernel debug options and looking more closely to see whether any of these kernel debug options have regressed upstream causing the performance to no longer be acceptable for the libguestfs testing and whether some of these debug options are still worth the cost for Rawhide testing.