Benchmarking The AMD EPYC Speed Boost Coming To Linux 5.18, Thanks To Scheduler/NUMA Improvement

Earlier this month I noted a Linux scheduler change queued into sched/core ahead of the Linux 5.18 cycle that is expected to help AMD EPYC processors and other select Zen processors in various workloads. The change has been in the works for several months and is about adjusting the allowed NUMA imbalance when spanning multiple LLCs. I've now carried out some of my own benchmarks on EPYC hardware and indeed is further ratcheting up the Linux kernel performance.

The patch talks up nice performance gains... Now testing on my side.

This change queued into "sched/core" ahead of next month's Linux 5.18 merge window is not an AMD-specific change but one that happens to benefit Zen processors due to their cache layout. Mel Gorman who authored the patch explained, "[A kernel scheduler change from 2020] allowed an imbalance between NUMA nodes such that communicating tasks would not be pulled apart by the load balancer. This works fine when there is a 1:1 relationship between LLC and node but can be suboptimal for multiple LLCs if independent tasks prematurely use CPUs sharing cache. Zen* has multiple LLCs per node with local memory channels and due to the allowed imbalance, it's far harder to tune some workloads to run optimally than it is on hardware that has 1 LLC per node. This patch allows an imbalance to exist up to the point where LLCs should be balanced between nodes."

Can confirm great performance uplift on EPYC 7003 series across a variety of workloads with the latest TIP sched/core changes.

Mel's own benchmarks when working on this patch saw up to a 272% improvement for the Stream memory benchmark and also other sizable wins like 10% better performance in Coremark and a maximum up to 17%, SPECjbb Java performance was up as much as 18%, NPB's embarrassingly parallel benchmark was around 17% faster, and other notable wins. Given the very promising results reported, I've been carrying out some of my own tests locally and the numbers I am seeing are equally exciting -- especially for this Linux kernel scheduler change just being around 50 lines of new code!

First up are some benchmarks ran with an AMD EPYC 75F3 2P server built around an ASRockRack ROME2D16-2T motherboard and running Ubuntu 21.10. The performance was compared between Linux 5.17 Git and then sched/core Git having the "sched/fair: Adjust the allowed NUMA imbalance when SD_NUMA spans multiple LLCs" patch in question. The same kernel Kconfig between both kernels was used.