NVIDIA Improving Linux NUMA Distance Interface To Enhance Performance

Written by Michael Larabel in NVIDIA on 19 July 2022 at 12:42 PM EDT. 2 Comments
NVIDIA engineers have been working on NUMA distance metrics within the Linux kernel to replace the simple local/remote NUMA preference interface currently used by some drivers for NUMA-aware memory allocations. In their testing this improved NUMA distance handling is leading to "significant performance implications" for throughput and CPU utilization.

This NVIDIA work isn't part of their graphics driver effort but rather on the Mellanox networking side of the house. Tariq Toukan summed it up in the latest spin of these kernel patches:
Implement and expose CPU spread API based on the scheduler's sched_numa_find_closest(). Use it in mlx5 and enic device drivers. This replaces the binary NUMA preference (local / remote) with an improved one that minds the actual distances, so that remote NUMAs with short distance are preferred over farther ones.

This has significant performance implications when using NUMA-aware memory allocations, improving the throughput and CPU utilization.

So far the results with the Mellanox and ENIC network drivers are looking very good from NVIDIA's testing on AMD EPYC 7763 servers:

See this patch series for more details.
Related News
About The Author
Michael Larabel

Michael Larabel is the principal author of Phoronix.com and founded the site in 2004 with a focus on enriching the Linux hardware experience. Michael has written more than 20,000 articles covering the state of Linux hardware support, Linux performance, graphics drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org automated benchmarking software. He can be followed via Twitter, LinkedIn, or contacted via MichaelLarabel.com.

Popular News This Week