Intel's One Line Of Linux Code For Speeding-Up Sapphire Rapids On Ubuntu
Recently I noticed out-of-the-box on Ubuntu Linux the performance of Intel Xeon Scalable "Sapphire Rapids" processors was much improved for some workloads compared to tests done just weeks ago on the same Sapphire Rapids server. It ended up being an issue coming full-circle and ultimately boils down to one line of code added within the Linux kernel.
When running Linux 6.3 on existing Ubuntu Linux installations or when moving to newer official Ubuntu Linux kernel image builds on 22.04/22.10/23.04, the Intel Xeon Scalable Sapphire Rapids performance has improved. After investigating, the issue came down to a change introduced during the Linux 6.3 kernel cycle and also back-ported by Canonical to existing Ubuntu Linux releases.
What was happening? This patch to adjust the balance_performance energy performance preference (EPP) for Sapphire Rapids. The onle line of code change for Sapphire Rapids is a big help for distributions like Ubuntu Linux that make use of the P-State "powersave" governor by default.
This performance fix by Intel ended up being motivated by an earlier Phoronix article: CentOS Stream & Clear Linux Achieve Greater Performance On 4th Gen Xeon Scalable Sapphire Rapids, EPYC Genoa. In that article from February I showed the Xeon Platinum 8490H performance across a variety of distributions, including Ubuntu Linux both with its default P-State "powersave" governor and switching to the performance governor as used by default on CentOS/RHEL and other Linux distributions.
From there Intel Linux engineer Srinivas Pandruvada investigated and came up with the aforementioned patch, in which he explained:
While the majority of server OS distributions are deployed with the "performance" governor as the default, some distributions like Ubuntu use the "powersave" governor by default. While using the "powersave" governor in its default configuration on Sapphire Rapids systems leads to much lower power, the performance is lower by more than 25% for several workloads relative to the "performance" governor. A 37% difference has been reported by www.Phoronix.com . This is a consequence of using a relatively high EPP value in the default configuration of the "powersave" governor and the performance can be made much closer to the "performance" governor's level by adjusting the default EPP value. Based on experiments, with EPP of 0x00, 0x10, 0x20, the performance delta between the "powersave" governor and the "performance" one is around 12%. However, the EPP of 0x20 reduces average power by 18% with respect to the lower EPP values. [Note that raising min_perf_pct in sysfs as high as 50% in addition to adjusting EPP does not improve the performance any further.] For this reason, change the EPP value corresponding to the the default balance_performance setting for Sapphire Rapids to 0x20, which is straightforward, because analogous default EPP adjustment has been applied to Alder Lake and there is a way to set the balance_performance EPP value in intel_pstate based on the processor model already. The goal here is to limit the mean performance delta between the "powersave" governor in the default configuration and the "performance" governor for a wide variety of server workloadsto to around 10-12%. For some bursty workloads, this delta can be still large, as the frequency ramp-up will still lag when the "powersave" governor is in use irrespective of the EPP setting, because the performance governor always requests the maximum possible frequency. Link: https://www.phoronix.com/review/centos-clear-spr/6 # 
That patch was picked up for the mainline Linux 6.3 cycle as a late merge window change. Surprisingly I hadn't noticed at the time this important one-liner being merged.
In turn this fix was back-ported to Ubuntu Linux kernel releases for the Jammy / Kinetic / Lunar kernels (Ubuntu 22.04 LTS / 22.10 / 23.04). The Ubuntu kernel builds with this P-State change being back-ported began rolling out to Ubuntu Linux users in recent weeks.
This behavior change though as a reminder is just when using Ubuntu Linux with its default powersave governor configuration. If you are already manually switching over to the P-State performance governor or using a Linux distribution that defaults to the performance governor, this change won't affect your Sapphire Rapids performance. Also, keep in mind that for my CPU server reviews and the like I am testing all CPUs already in the "performance" governor mode so this change ultimately doesn't impact previously shown review results of Sapphire Rapids vs. AMD EPYC Genoa and similar where all were running in the performance governor...
I'd still love to see Canonical switch over to using the performance governor on Ubuntu Server installations, but that's a separate matter... In any event, for those curious about what this means for Intel Sapphire Rapids performance when running with powersave / out-of-the-box on Ubuntu Linux, I ran some comparison benchmarks.