Benchmarking Linux With The Retpoline Patches For Spectre

[Michael]

(cc CtrlAltShift ) I take this whole article as: comparisons cannot be made across CPUs, you only compare the relative performance of a CPU vs itself with and without the patches.

For instance, the different systems use different drives, so it's not really comparable. They also use wildly different sizes of RAM. I am not saying that AMD or intel should be better than each other, I'm taking no positions at all. Just saying that the interpretations should be: "AMD's epyc was x% faster/slower, intel's offer was y% faster/slower", but you can't say that either was better than the other (based on how this article is set up).

Michael, am I understanding this properly? If yes, this is not a criticism of the article at all - just trying to clarify how to interpret the results. Cheers!

Right this testing is intended at just comparing Retpoline and not different CPUs against their performance.

[duby229]

(cc CtrlAltShift ) I take this whole article as: comparisons cannot be made across CPUs, you only compare the relative performance of a CPU vs itself with and without the patches.

For instance, the different systems use different drives, so it's not really comparable. They also use wildly different sizes of RAM. I am not saying that AMD or intel should be better than each other, I'm taking no positions at all. Just saying that the interpretations should be: "AMD's epyc was x% faster/slower, intel's offer was y% faster/slower", but you can't say that either was better than the other (based on how this article is set up).

Michael, am I understanding this properly? If yes, this is not a criticism of the article at all - just trying to clarify how to interpret the results. Cheers!

ah yeah, ok that explanation makes perfect sense. Thanks.

[linuxgeex]

Koolaid doesn't inhibit your thought process. Perhaps you should lay off the alcohol?

Not defending the ad-hominem by the original poster, but I'd just like to clarify that the Koolaid did in fact inhibit your thought process, since it was lethal.

[Wielkie G]

My anger is against who tried to minimize the performance impact with such stupid slides.

The "negligible performance impact" is related to variant 1. The retpoline patchset fixes variant 2, which AMD deemed is impossible on their CPUs.

Not sure what is the plan forward. Either retpoline won't be enabled on AMD CPUs (since, as of now, it is not needed) or it will (just in case someone reproduces variant 2 on AMD CPUs). Since variant 2 depends on branch prediction inner workings, it is heavily dependent on CPU microarchitecture. The fact it was not done on AMD CPUs doesn't mean it's impossible. For now, without retpoline, we have something like "security by obscurity" - AMD CPUs are safe since their branch predictor was not reverse-engineered yet.

[dillon]

RetPoline is only partially effective for SkyLake or later CPUs. You also need to test with the Intel microcode patched and the IBRS feature turned on. Test with IBRS mode 1 and IBRS mode 2 (on Linux). IBRS mode 1 is kernel protection (IBRS enabled only while in the kernel). IBRS mode 2 is kernel & user protection (IBRS enabled at all times). Mode 2 will result in a 30-40% loss in performance. What people are most interested in right now is mode 1 operation... how much performance loss is there when only the kernel is protected.

RetPoline itself isn't expected to add any significant overhead because only 1 in every 100 calls is typically going to be an indirect call (which is what RetPoline patches).

-Matt

[dillon]

Let me be a little more clear.

RetPoline is effective for all Intel CPUs that are older than SkyLake. For SkyLake and later CPUs, RetPoline is only partially effective.

The microcode update is not intended to replace RetPoline. Its intended to work in conjunction with RetPoline, and it is not intended to be left on (linux IBRS mode 2), though some people will be paranoid to leave it on and take the 30-40% performance hit. For older CPUs, even with the presence of the microcode update, it might be better to just rely on RetPoline and not enable IBRS due to the extreme performance penalty. Intel has stated on the LKML lists that the microcode will get better over time (probably mean over the next few years, honestly), and ultimately can just be left enabled full-time, but that time is not now.

AMD is also working on microcode updates for Spectre, but I have no information on what it does or the ABI. One thing it does NOT do is disable branch speculation (lots of press articles say it does, but it does not). The effectiveness of RetPoline on AMD systems is not known, either.

At the moment, all of these mitigations are poor hacks. Also, I would be careful of any wordage (from both Intel and AMD) about mitigations being solved by software. That will never happen. Bleeding edge recompiled software, sure. Kernels... probably (and eventually gain penetration). But probably almost none of the application software out in the wild will ever get these hacks.

-Matt

[dillon]

This gives us some estimate on performance cost of retpolines, but keep in mind that they do not mitigate Spectre for Skylake and later CPUs. Intel introduced some optimizations in this microarchitecture, which enable speculation for 'ret' instructions.
Skylake, Kaby Lake and Coffee Lake, require a microcode update to fix Spectre (at least the "Branch Target Injection" part). This will allow the kernel to set two flags in the CPU - IBRS and IBPB. As far as I know, 4.14.12 and 4.15-rc7 releases do not contain appropriate patches yet. Apparently RedHat already has relevant patches in place. [1]

The "good" news is that the microcode patch should protect from Spectre without rebuilding whole userland with retpolines. At least in Xen, the cheapest variant of retpolines is automatically selected in the kernel when Skylake (or later) is detected. [2]
The bad news is that apparently IBRS also has a serious performance penalty.

So keep in mind that the results for 8700K and 7980XE will change once a complete solution becomes available.

[1] https://access.redhat.com/articles/3311301
[2] https://xenbits.xen.org/gitweb/?p=pe...ions-v6.5#l232

Ah, sorry to burst your bubble, but that isn't really the case. Enabling IBRS for the kernel will be bad enough, enabling IBRS for userland too will completely destroy performance. I doubt that any OS will be enabling IBRS for usermode by default. If you run Linux, IBRS for usermode + kernel is IBRS mode 2 operation. Kernel-only is IBRS mode 1 operation.

-Matt

[decep]

I wonder if Amazon or some other large cloud provider could sue Intel and AMD bigtime for performance loss due to gross incompetence. You know just have them charge these diffs in profits until they stop using Intel/AMD in the far future.

From the cloud provider perspective, this will only increase revenue. Higher CPU utilization for existing workloads means customers will pay for more resources.

[naur]

The microcode update is not intended to replace RetPoline. Its intended to work in conjunction with RetPoline

I didn't know that retpolines are still required with IBRS. I've thought that setting the MSR disables prediction for existing indirect jmp instructions. Which would still be bad performance-wise, but at least it could mitigate Spectre in software that can't be recompiled with retpolines.

I'm just hoping that future kernel patches will allow setting IBRS=2 in a specific cgroup (running only vulnerable software, like browsers).

[darkbasic]

The "negligible performance impact" is related to variant 1. The retpoline patchset fixes variant 2, which AMD deemed is impossible on their CPUs.

Not sure what is the plan forward. Either retpoline won't be enabled on AMD CPUs (since, as of now, it is not needed) or it will (just in case someone reproduces variant 2 on AMD CPUs). Since variant 2 depends on branch prediction inner workings, it is heavily dependent on CPU microarchitecture. The fact it was not done on AMD CPUs doesn't mean it's impossible. For now, without retpoline, we have something like "security by obscurity" - AMD CPUs are safe since their branch predictor was not reverse-engineered yet.

The fix for variant 2 is called Retpoline while the fix for variant 3 (Meltdown) is called KPTI (kernel page table isolation). How is called the fix for variant 1?