Linux 5.16's New Cluster Scheduling Is Causing Regression, Further Hurting Alder Lake

Long story short, the x86 cluster scheduling new in Linux 5.16 is the culprit. Bisecting the kernel regression led precisely to the introduction of that x86 cluster-aware scheduling support.

That Linux CPU cluster scheduler work originated with Arm server work for SoCs with clusters of cores per NUMA node. It was then adapted for x86 support to help platforms like Intel's Jacobsville with clusters of cores having shared L2 cache. As of last testing, this cluster scheduling doesn't have any impact for AMD processors with not yet having any topology support there.

SCHED_CLUSTER is on-by-default with its Linux 5.16 introduction.

With Intel Alder Lake now with the hybrid architecture and the mix of P and E cores, the cluster-aware scheduling ia a set-back as presumably there is no recognition/emphasis on the core differences. So with Linux 5.16 it's likely shifting these benchmarks now to running on the E cores instead as being part of a different cluster, not knowing it's detrimental to the performance in doing so.

The cluster layout for the Core i9 12900K with Linux 5.16 Git.

With SCHED_CLUSTER defaulting to enabled by default, this means Intel Alder Lake on Linux 5.16 right now is seeing reduced performance compared to Linux 5.15 as demonstrated in this article and confirmed by the bisecting. It will be interesting to see if there is a remedy for Linux 5.16 over the next two months of release candidates and what that remedy will be, short of disabling the cluster scheduling feature. Presumably Intel Thread Director integration would benefit/help the situation but so far we haven't seen any patches there and now too late for the 5.16 merge window.

At least for those not on Alder Lake, with these workloads I haven't found Linux 5.16 regressing outside of this context. But back now to looking at some other unrelated Linux 5.16 performance changes I'm still investigating with other hardware... Those that appreciate the relentless effort I do around Linux hardware can join Phoronix Premium or make a PayPal tip to help in these often very time consuming and energy-hungry testing.