Linux 5.15 Addressing Scalability Issue That Caused Huge IBM Servers 30+ Minutes To Boot
Written by Michael Larabel in Linux Kernel on 1 September 2021 at 04:00 PM EDT. 31 Comments
LINUX KERNEL --
Very large IBM mainframes/servers were taking 30+ minutes to boot the Linux kernel... No, just not for POST'ing the system with memory training and the like, but for loading Linux. Fortunately, with the Linux 5.15 kernel there is a set of scalability enhancements to allow these large IBM systems to be able to boot in around five minutes.

With the driver core changes for Linux 5.15 is a set of patches working on enhancing the performance of Kernfs for functionality used around pseudo file-systems like sysfs. Leading to this Kernfs locking and concurrency improvements were engineers finding that large IBM Power mainframe systems with "several hundred CPUs and 64TB of RAM" were taking 30+ minutes to just boot the Linux kernel. Extra kernel parameters were also needed to avoid the kernel timing out on boot.

The extremely long boot times for modern, high-end servers was found to be the result of many path look-ups for non-existent files and extreme locking contention within the VFS code.

Making matters worse is that with the 64TB of system memory and IBM Power dividing them into 256MB local blocks exposed via sysfs, a heck of a lot of sysfs nodes are being created.

With the Kernfs scalability improvements for benefiting sysfs that are in the driver core changes for Linux 5.15, these IBM systems can go from 30+ minutes to boot to now under five minutes. The changes involve switching Kernfs mutex to using a read-write semaphore for allowing node searches in parallel, improving path resolution, and using the VFS negative dentry caching.

These Kernfs improvements and more can be found via the driver core PR that was merged today for the 5.15 merge window.
Related News
About The Author
Author picture

Michael Larabel is the principal author of Phoronix.com and founded the site in 2004 with a focus on enriching the Linux hardware experience. Michael has written more than 20,000 articles covering the state of Linux hardware support, Linux performance, graphics drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org automated benchmarking software. He can be followed via Twitter or contacted via MichaelLarabel.com.

Popular News This Week