Ubuntu Looking At Applying Low-Latency Optimizations To Its Generic Kernel

Ubuntu has long provided a "low-latency" kernel build intended for industrial embedded systems and other latency sensitive environments. Ahead of Ubuntu 24.04 LTS, Canonical is looking at applying those low-latency optimizations to their generic kernel build.

Ubuntu 24.04 LTS keeps getting more interesting from the performance department... From being aggressive in targeting the Linux 6.8 kernel, frame pointers by default, experimenting with Ubuntu x86-64-v3 builds, and more, this next long-term support release of Ubuntu Linux is looking quite interesting. The latest is now considering applying their low-latency changes to the generic kernel build used by Ubuntu.

Ubuntu's low-latency kernel is mostly Kconfig configuration changes applied to their kernel build that they are now at least considering to make by default. This Launchpad bug report elaborates:

Ubuntu provides the "lowlatency" kernel: a kernel optimized for applications that have special "low latency" requirements.

Currently, this kernel does not include any specific UBUNTU SAUCE patches to improve the extra "low latency" requirements, but the only difference is a small subset of .config options.

Almost all these options are now configurable either at boot-time or even at run-time, with the only exception of CONFIG_HZ (250 in the generic kernel vs 1000 in the lowlatency kernel).

Maintaining a separate kernel for a single config option seems a bit overkill and it is a significant cost of engineering hours, build time, regression testing time and resources. Not to mention the risk of the low-latency kernel falling behind and not being perfectly in sync with the latest generic kernel.

Enabling the low-latency settings in the generic kernel has been evaluated before, but it has been never finalized due to the potential risk of performance regressions in CPU-intensive applications (increasing HZ from 250 to 1000 may introduce more kernel jitter in number crunching workloads). The outcome of the original proposal resulted in a re-classification of the lowlatency kernel as a desktop-oriented kernel, enabling additional low latency features (LP: #2023007).

Among the changes in the low-latency kernel are enabling CONFIG_NO_HZ_FULL for full tickless mode, CONFIG_RCU_NOCB_CPU for moving RCU callbacks to the KThread context, CONFIG_RCU_LAZY for batch RCU callbacks, and CONFIG_HZ=1000 for 1000Hz to improve system responsiveness.

The worry is primarily over risking HPC / server workload performance. But, hey, I invite that as an opportunity now to run some fresh Phoronix benchmarks of the Ubuntu kernel offerings in the coming days...

The matter of applying the low-latency settings to Ubuntu's generic kernel build are still to be determined and those wanting to find the latest can do so via this Launchpad bug. It will be interesting to see what they decide and what other performance changes they may pursue for Ubuntu 24.04 LTS.

Update: Here are some generic vs. low-latency kernel benchmarks for desktop and HPC workloads.