Highly Threaded Linux Software Running Under CFS Quotas See Big Performance Fix
Written by Michael Larabel in Linux Kernel on 15 October 2019 at 02:00 PM EDT. 11 Comments
LINUX KERNEL --
Thanks to a Linux kernel fix that is likely to be back-ported to the various stable series, highly threaded software running under CFS quotas for enforcing CPU limits are about to be much faster. At least in a synthetic test case, the kernel fix yields a 30x improvement in performance.

Spotted by the Kubernetes community but affecting others with highly threaded workloads and making use of a CFS quota to restricted shared CPU resources, it turns out that highly-threaded applications are routinely not getting "their fair share" of the CPU, leading to lower than expected performance and higher latency.

This has been a known bug for more than one year and a kernel bug report on unexpected CFS throttling since late 2017. The issue is believed to be recently fixed up for mainline Linux 5.4 and pending for back-ports after the patch was volleyed around the kernel mailing list for a few months.

There is the fix that is a few dozen lines of code that removes the expiration of CPU-local slices:
It has been observed, that highly-threaded, non-cpu-bound applications running under cpu.cfs_quota_us constraints can hit a high percentage of periods throttled while simultaneously not consuming the allocated amount of quota. This use case is typical of user-interactive non-cpu bound applications, such as those running in kubernetes or mesos when run on multiple cpu cores.
...
This greatly improves performance of high-thread-count, non-cpu bound applications with low cfs_quota_us allocation on high-core-count machines. In the case of an artificial testcase (10ms/100ms of quota on 80 CPU machine), this commit resulted in almost 30x performance improvement, while still maintaining correct cpu quota restrictions.

Thanks to Phoronix reader Mark for pointing out this recent kernel change.
Related News
About The Author
Author picture

Michael Larabel is the principal author of Phoronix.com and founded the site in 2004 with a focus on enriching the Linux hardware experience. Michael has written more than 20,000 articles covering the state of Linux hardware support, Linux performance, graphics drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org automated benchmarking software. He can be followed via Twitter or contacted via MichaelLarabel.com.

Popular News This Week