VMware: ESXi VM Performance Tanks Up To 70% Due To Intel Retbleed Mitigation

[oiaohm]

I still think it's not that funny
Imagine you have a farm of servers with intel processors. You bought them because of their performance, probably due to very good price to performance ratio.
Suddenly that changes and those processors are a very bad deal. You would feel a bit tricked because that is purely fault of the company and there is not really an alternative, except exposing yourself to security issues, which again, are things you didn't know in advance that are part of the deal.

Are refunds possible based on these regressions? "Well, duh, yes" would be a good answer to this one

The bad news here is with CPU defects most cases as long as the CPU will still work the same as what it did when you bought it you can not get a refund. It you who want a secure system.

Please note this x86 cpu vs a mainframe cpu. Mainframe you renting the system so if the cpu in mainframe is defective the maker has to replace it at their cost.

x86 servers are basically the cheap choice with no on going payments. When it comes to security with x86 you are basically buying it as is where is so sometime you get bitten. Yes with the recent x86 cpu design faults people have been bitten a lot..

.

[archkde]

I still think it's not that funny
Imagine you have a farm of servers with intel processors. You bought them because of their performance, probably due to very good price to performance ratio.
Suddenly that changes and those processors are a very bad deal. You would feel a bit tricked because that is purely fault of the company and there is not really an alternative, except exposing yourself to security issues, which again, are things you didn't know in advance that are part of the deal.

Are refunds possible based on these regressions? "Well, duh, yes" would be a good answer to this one

I don't know the situation where you live, but at least in most of Europe, refunds are only possible within the first 2 years after purchase. Given that this is SKL, you're out of luck in any case, even if the mitigation impact was reason enough for a refund. (I rather think that the mitigations would legally count as a repair of the security vulnerabilities, though, so no refund would be possible in the first place.)

[ll1025]

It's worth noting here, in the context of the Linux vs Windows benchmarks over the last several years-- Windows has been running with these mitigations enabled *for years*.

It gives me no joy to see 5.19 performance tank but the calls to disable the mitigations are absurd; if Windows (of all things) can achieve normalcy with the mitigations, then so can Linux. Security has performance costs-- get over it.

On a side note, I hope the disastrous performance implications are not the reason we have not seen 5.19-to-windows or Ubuntu 22.04.1-vs-Windows benchmarks on phoronix. We all know what it's going to look like but it would be nice to see them, even if only by way of apology for the last few years of slanted mitigations=off benchmarks.

[oiaohm]

It gives me no joy to see 5.19 performance tank but the calls to disable the mitigations are absurd; if Windows (of all things) can achieve normalcy with the mitigations, then so can Linux. Security has performance costs-- get over it.

https://techcommunity.microsoft.com/t5/virtualization/hyper-v-hyperclear-retbleed-update/ba-p/3576717

The reality is also Microsoft has areas they did not do correctly years ago as well that can be on desktop computers. Yes using WSL2 has taken a bit of a performance hit from the hyper-v update.

Problem when you have choice of methods to fix a problem you can pick the wrong one.

https://www.phoronix.com/review/rembrandt-windows-linux/2

Yes there are benchmarks of 5.19 vs Windows and ubuntu 22.04. Windows mitigations are not cheap either.

Notice here the bare metal benchmarks with Linux none of them showed a 70% loss. There are losses but not that much. Also microsoft hyper-v dealing with retbleed has also taken a performance hit.

There is a question is the problem in the ESXi hypervisor based on how it implemented that the retbleed patch just brings to floor.

[LightBit]

I rather think that the mitigations would legally count as a repair of the security vulnerabilities, though, so no refund would be possible in the first place.

Mitigations are mostly like workarounds not repair.

[ll1025]

https://techcommunity.microsoft.com/...e/ba-p/3576717
The reality is also Microsoft has areas they did not do correctly years ago as well that can be on desktop computers. Yes using WSL2 has taken a bit of a performance hit from the hyper-v update.
Problem when you have choice of methods to fix a problem you can pick the wrong one.

I'm not clear how the link you provide backs up your claim. They're saying that the existing mitigations in Hyper-V (and presumably WSL2) already solve Retbleed-- because they implemented IBRS years ago while Linus rejected the patchsets as unnecessary and too expensive. It sounds like Hyper-V goes beyond that and performs VM core isolation for further protection (as well as SEV / FME for other threats). Can you clarify how you think the article supports the idea that MS did not do it correctly?

https://www.phoronix.com/review/remb...indows-linux/2
Yes there are benchmarks of 5.19 vs Windows and ubuntu 22.04. Windows mitigations are not cheap either.

First-- thanks for finding this, I don't know how I missed it. I was specifically looking for these benchmarks for the weeks around 5.19 and 22.04.1 and thought it was an intentional lapse; glad to see that I am incorrect.

But it is hard to call those apples-to-apples when Windows is running with VBS / HVCI / MBEC for which there are no equivalents on Ubuntu 22.04, and which are *known* to be rather expensive mitigations. Casual googling suggests 10-20% perf loss on recent CPUs, but not a lot of information on its Ryzen impact or the impact on these gpu benchmarks.

It would be nice to see comparisons that attempt to measure the same thing; obviously there are going to be feature differences (SELinux / apparmor vs "exploit guard", LUKS vs bitlocker) but comparing what is essentially a completely virtualized Windows (VBS) to a bare-metal Linux is never going to give meaningful output. I certainly dont think you could call VBS / HVCI a default configuration.

Notice here the bare metal benchmarks with Linux none of them showed a 70% loss. There are losses but not that much. Also microsoft hyper-v dealing with retbleed has also taken a performance hit.

No doubt. My understanding is that VMWare's benchmark is pretty narrow and could be construed as a worst-case.

[ll1025]

Mitigations are mostly like workarounds not repair.

This is silly.

ASLR / DEP / FME / the entire acronym soup are all "mitigations" for situations where a security boundary is violated by a hardware or software flaw. There will never be an end to hardware/software flaws, and patching hardware flaws is rather untenable. Speculative execution provides enough of a performance boost that it is worth keeping them with mitigations in place.

The actual "fix" would probably be to disable speculative execution, but I don't know of anyone who wants that. If you think the cost of IBRS is high, wait till you see the impact of disabling out-of-order execution.

[oiaohm]

I'm not clear how the link you provide backs up your claim. They're saying that the existing mitigations in Hyper-V (and presumably WSL2) already solve Retbleed-- because they implemented IBRS years ago while Linus rejected the patchsets as unnecessary and too expensive. It sounds like Hyper-V goes beyond that and performs VM core isolation for further protection (as well as SEV / FME for other threats). Can you clarify how you think the article supports the idea that MS did not do it correctly?

Hyper-v code does not do IBRS stuff either. HyperClear is not IBRS. IBRS was implemented in the windows kernel but Hyper-v does not. The reality here is IBRS is too heavy for Microsoft hyper-v as well. Its not just Linus who rejected IBRS for being too expensive.

Retpoline is developed solution for Linux to the speculative execution issues. IBRS is the intel microcode based solution to the speculative execution issue. HyperClear is Microsoft solution for the hyper-v. Windows solution in kernel also is not IBRS.

The reality here is IBRS is basically lemon designed by Intel. Retpoline turns out to be less of a lemon still has issues. HyperClear and other solutions by Microsoft have also seen updates over the time all costing different levels of performance. HyperClear is kind of in it name it clearing memory all the time and doing this when it not really required does cause performance overhead.

Please note 70% overhead of the Retbleed fix is nothing compared to the IBRS overhead.

https://nationalcybersecuritynews.today/retbleed-call-depth-tracking-mitigation-eyed-to-avoid-ibrs-performance-horror-show-linux-linuxsecurity/

IBRS is still over double the overhead of the RetBleed fixes.

The reality here is Microsoft developers and Linux Developers have both rejected IBRS as solution to the problem.

[ll1025]

Hyper-v code does not do IBRS stuff either. HyperClear is not IBRS. IBRS was implemented in the windows kernel but Hyper-v does not. The reality here is IBRS is too heavy for Microsoft hyper-v as well. Its not just Linus who rejected IBRS for being too expensive.

Retpoline is developed solution for Linux to the speculative execution issues. IBRS is the intel microcode based solution to the speculative execution issue. HyperClear is Microsoft solution for the hyper-v. Windows solution in kernel also is not IBRS.

This is (mostly) incorrect.

1. When you enable Hyper-V, all of windows becomes a guest VM so it's hairsplitting in the most irrelevant way to say "HyperClear....was not implemented in the windows kernel". It's implemented at ring 0 or -1, and most people would call that the kernel.

2. Windows does implement IBRS, which is why bare metal is not impacted by retbleed:

Note that Windows systems are not affected given that these systems use Indirect Branch Restricted Speculation (IBRS) by default which is is also the mitigation being made available to Linux users

3. The HyperClear mitigation is not "clear memory all the time". It generally ensures that VM vCPUs are scheduled to the same virtual processors on the same cores and then only flushes memory associated with a particular virtual processor.

4. eIBRS has a lower performance hit than retpolines, which is why prior to the March VUSec research eIBRS was the default for processors that supported it and enabling it in kernel would disable retpolines on Linux.

5. 5.19 literally just implemented IBRS and Windows has had it for ~4 years, so to say "Microsoft developers and Linux Developers have both rejected IBRS" is a truly amazing claim.

6. In fact there were patchsets going back to 2018 for IBRS and SUSE implemented them downstream. *Some* developers (mainly: Linus) rejected IBRS but it's now apparent that it is necessary and Linus should be eating some humble pie-- especially for implying that this is an Intel-only issue despite affecting essentially every modern, performant processor.