Intel Fixing Up Sub-NUMA Clustering For Linux So That It Behaves With RDT

Written by Michael Larabel in Intel on 11 September 2023 at 02:44 PM EDT. 5 Comments
Sub-NUMA Clustering with Intel Xeon processors allows for splitting up the CPU cores, cache, and memory into multiple NUMA domains for enhancing the performance of NUMA-aware applications. While SNC can help in a number of cases especially plenty of HPC and server workloads, currently it's not properly supported if making use of Resource Director Technology (RDT) on modern Intel CPUs. That is in the process of changing with new Linux kernel patches being worked on by Intel.

Intel Resource Director Technology allows for insight (monitoring) and control over shared resources such as memory bandwidth and last-level cache use by applications and VMs. But these extra insights and controls around cache and memory allocation clash with Sub-NUMA Clustering. So the Linux kernel patches being worked on by Intel engineers are working to allow RDT and SNC to better co-exist, although RDT may still not provide the most accurate information on an SNC-enabled server. This has been a known problem being worked on for months by Linux engineers.

Intel Sapphire Rapids CPU

The latest patches by Intel engineer Tony Luck explain:
"The Sub-NUMA cluster feature on some Intel processors partitions the CPUs that share an L3 cache into two or more sets. This plays havoc with the Resource Director Technology (RDT) monitoring features. Prior to this patch Intel has advised that SNC and RDT are incompatible.

Some of these CPU support an MSR that can partition the RMID counters in the same way. This allows for monitoring features to be used (with the caveat that memory accesses between different SNC NUMA nodes may still not be counted accuratlely.

Note that this patch series improves resctrl reporting considerably on systems with SNC enabled, but there will still be some anomalies for processes accessing memory from other sub-NUMA nodes."

These patches are still being reviewed but perhaps they'll be ready for mainline by the v6.7 kernel cycle later this year.
Related News
About The Author
Michael Larabel

Michael Larabel is the principal author of and founded the site in 2004 with a focus on enriching the Linux hardware experience. Michael has written more than 20,000 articles covering the state of Linux hardware support, Linux performance, graphics drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and automated benchmarking software. He can be followed via Twitter, LinkedIn, or contacted via

Popular News This Week