Benchmarking Linux With The Retpoline Patches For Spectre

While the Kernel Page Table Isolation (KPTI) patches were quick to land in the mainline Linux kernel for addressing the Meltdown CPU vulnerability, the "Retpoline" patches are still being worked on as the leading approach on the Linux side for dealing with the Spectre CPU vulnerability. The Retpoline patches are said to have little impact on performance, but here are our benchmarks of these kernel patches for seeing how they affect a variety of AMD and Intel systems.

Retpoline is the Google-led approach for dealing with Spectre or "speculative execution" and preventing branch target injection attacks by avoiding speculative indirect branches within the kernel code. Google has said Retpoline has a "negligible impact on performance" and is already being used on all of their systems and is also found in use already by some other vendors. But as of writing the code has yet to hit the mainline Linux kernel. This weekend patches were posted for Retpoline v5 and things are getting into shape for merging but it's still not immediately clear if they will try to get the patches in time for Linux 4.15, if they will wait until the Linux 4.16 merge window, and/or if they will try to land it in 4.16 and then back-port it to current stable series.

For maximum impact, the Linux kernel with Retpoline patches also needs to be built against a patched version of GCC (or the LLVM Clang) compiler. There are GCC and LLVM patches pending for targeting Spectre and introducing the new -mindirect-branch, -mindirect-branch-loop, -mfunction-return, and -mindirect-branch-register switches. But those compiler patches as of writing have yet to be mainlined. But even without a compiler supporting -mindirect-branch=thunk-extern, the Retpoline kernel patches will eliminate indirect branches in Assembler code, just not "full protection."

With the patches, the support is toggled via the CONFIG_RETPOLINE Kconfig switch and will be defaulting to enabled for all architectures and CPUs. Yes, this includes AMD CPU support but on current AMD architectures the Retpoline is simpler. On Retpoline-enabled kernels, users are able to boot the kernel with the "noretpoline" boot parameter to disable the protection.

For our initial benchmarks of the yet-to-be-merged Retpoline patches, I tested the v5 patch-set on several systems this week in the below configurations:

noretpoline - The Linux 4.14-based with Retpoline patches maintained by David Woodhouse as of their v5 state as of Saturday morning, but booting the system with "noretpoline". These results basically show the performance without Retpoline.

Retpoline - The Linux 4.14-based Retpoline-patched kernel booted with Retpoline present. This kernel though was built with Ubuntu 17.10's stock GCC 7.2 compiler, which doesn't contain the Spectre patches / "mindirect-branch" support. So this build is only of limited effectiveness but is what users will find without an upgraded compiler with the yet-to-land Spectre code changes.

Retpoline + GCC - The same Linux 4.14 kernel branch with Retpoline patches but when built using David Woodhouse's GCC 7.2 branch that does contain the patches as of this weekend. This run shows the impact with full protection for Spectre / speculative execution.

I tested these three kernel configurations on a range of AMD and Intel systems with distinctly different hardware from low-end to ultra high-end including:

- Intel Core i3 7100
- Intel Core i7 8700K
- Intel Core i7 7980XE
- 2 x Intel Xeon Gold 6138
- AMD Ryzen 7 1800X
- AMD EPYC 7601

This should provide a pretty broad look at any performance impact from the Linux Retpoline protection. Note that these kernels do have the already-merged KPTI support present and enabled for Intel CPUs for Meltdown protection. A range of benchmarks were run via the open-source Phoronix Test Suite benchmarking software. As the Linux Retpoline support matures, of course, there will be new tests on Phoronix