Toggling Spectre Mitigations On Xeon Scalable Ice Lake Show Little Runtime Difference

As usual when getting my hands on a new processor family, I was curious about the performance difference if booting the Xeon Platinum 8380 "Ice Lake" processors with the Spectre security mitigations disabled at run-time. Ultimately there was very little difference when using the standard "mitigations=off" option for these new Intel server processors.

Intel 3rd Gen Xeon Scalable "ice Lake" processors are not affected by Meltdown, MDS, L1TF, ITLB Multihit, SRBDS, or TAA, but do still require kernel protections involving Spectre V1/V2/V4. Spectre V1 mitigation on the Ice Lake server CPUs involve usercopy/SWAPGS barriers and __user pointer sanitization, Spectre V2 on these new CPUs involves enhanced IBRS (Indirect Branch Restricted Speculation) and conditional IBPB (Indirect Branch Prediction Barrier) and RSB (Return Stack Buffer) filling.

Booting the Linux kernel with the mitigations=off option allows disabling these relevant Spectre protections at boot-time albeit putting the system vulnerable. With recent generations of Intel processors since addressing Meltdown and co with hardware architecture changes has lessened the impact of running with these software mitigations disabled while now with Ice Lake Xeon there is almost no run-time difference. That's not to say there are no internal performance costs associated with their hardware/architectural mitigations but in terms of what can be still toggled by the user if so choosing is now almost zeroed out with these brand new server processors.

When testing on the dual Xeon Platinum 8380 with Ubuntu 21.04 on a Linux 5.11 kernel booted by default/out-of-the-box with the relevant mitigations in place and then rebooting with "mitigations=off" to put the system in a vulnerable state but trying to juice out extra performance, there was now almost no measurable difference.

When running various workloads that in the past tended to show a difference with mitigations=off, on the Ice Lake server processors there wasn't much of a difference at all.

The side-by-side above just showing for the subset of workloads where there was even anything close to being noteworthy, all the other data points not shown for their flat performance but can be viewed on the OpenBenchmarking.org link below. In a few Java workloads there were some small shifts but nothing like we've seen on prior generations of Intel CPUs or even when it comes to AMD processors with their Spectre mitigations.

Those interested can see all the benchmark data on OpenBenchmarking.org but there isn't much in the way of a measurable difference of the workloads tested when toggling the software-controlled mitigations on new Ice Lake servers. Easily the least difference we've seen on modern x86_64 CPUs when running default vs. mitigations=off. Especially with production servers it is recommended to run the kernel in its default (mitigated) state for security while now these results show there is little to gain with these Ice Lake server processors from running with those controllable Spectre mitigations disabled.