NVIDIA Improving Linux NUMA Distance Interface To Enhance Performance

[phoronix]

Phoronix: NVIDIA Improving Linux NUMA Distance Interface To Enhance Performance

NVIDIA engineers have been working on NUMA distance metrics within the Linux kernel to replace the simple local/remote NUMA preference interface currently used by some drivers for NUMA-aware memory allocations. In their testing this improved NUMA distance handling is leading to "significant performance implications" for throughput and CPU utilization...

https://www.phoronix.com/scan.php?page=news_item&px=NVIDIA-NUMA-Distance-Metrics

[timofonic]

Anyone able to decipher this? Please...

[coder]

Anyone able to decipher this? Please...

Let's start with the patch comment quoted in the article:

Implement and expose CPU spread API based on the scheduler's sched_numa_find_closest(). Use it in mlx5 and enic device drivers. This replaces the binary NUMA preference (local / remote) with an improved one that minds the actual distances, so that remote NUMAs with short distance are preferred over farther ones.

This has significant performance implications when using NUMA-aware memory allocations, improving the throughput and CPU utilization.

So, it sounds like the kernel's scheduler has a more nuanced view of NUMA, in its sched_numa_find_closest() function. They're enabling device drivers to use the same (or similar) underlying model, rather than a simplistic local vs. remote model.

For machines with more than one level of hierarchy in their NUMA topology, this could be a win for locating resources closer to the device or perhaps the client userspace process than currently happens.

Lastly, as they note it's from the Mellanox side of the house, whose products I expect are predominantly used on machines with multi-level NUMA.