Linux 5.18's KVM Squeezes In AMD Nested Virtualization Improvements

[phoronix]

Phoronix: Linux 5.18's KVM Squeezes In AMD Nested Virtualization Improvements

Last week saw the main KVM virtualization pull for the Linux 5.18 feature updates while sent in today was a second batch of improvements for the Kernel-based Virtual Machine...

https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.18-More-KVM

[Vorpal]

I'm not a server kind of person. Could someone please enlighten me to the use cases of nested and double nested virtualisation? And how many layers down do people typically go?

[dragorth]

I'm not a server kind of person. Could someone please enlighten me to the use cases of nested and double nested virtualisation? And how many layers down do people typically go?

Well, if you are supporting many of the new Windows 10/11 features, from WSL2 to Android to some of the new security features, they require the ability to use hardware virtualization, which is what nesting enables. It is also used in Android development using VS. You don't have to use these features, but the security may be designated as a requirement for any windows VM in an enterprise environment, for an example of how this can creep in.
I don't know of any other use case that might make it to production, but I am sure there are many more that are used in testing.

[Vorpal]

Well, if you are supporting many of the new Windows 10/11 features, from WSL2 to Android to some of the new security features, they require the ability to use hardware virtualization, which is what nesting enables. It is also used in Android development using VS. You don't have to use these features, but the security may be designated as a requirement for any windows VM in an enterprise environment, for an example of how this can creep in.
I don't know of any other use case that might make it to production, but I am sure there are many more that are used in testing.

Thanks! That still sounds like one level of nesting though.

[smotad]

I use nested virtualization to set up labs on my laptop. The first level of virtual machines are simulating physical hosts. The nested virtual machines are simulating virtual machines on a production host. I think once I even went a further level.

[brent]

It's turtles all the way down.

[polarathene]

Thanks! That still sounds like one level of nesting though.

Those type of environments usually virtualize those hosts though, it's just modern setups use their own virtualization which in such an environment requires nested virtualization.

[kylew77]

Do any of the *BSD hypervisors support nested virtualization and what is so hard about implementing it? I learned on BSD NOW podcast that bhyve doesn't support it on FreeBSD and I'm pretty sure VMM doesn't support nested virtualization either. I don't know about NetBSD's hypervisor or DragonFly's.

[dragorth]

Thanks! That still sounds like one level of nesting though.

You have to sorta understand that the way Windows now works, bare Windows is essentially virtualized. That security I mentioned? It isn't for VMs, it is for the host. So any VMs running on such a host, such as the WSL/WSA, Android Dev VM etc are all nested under Windows. This started in the Win 8 era, and has become more comprehensive since.

Now, most of this stuff is still optional for Windows 10, but one of the reasons for Win11's higher CPU cut-off is so MS could enable some of it by default.

[dragorth]

Do any of the *BSD hypervisors support nested virtualization and what is so hard about implementing it? I learned on BSD NOW podcast that bhyve doesn't support it on FreeBSD and I'm pretty sure VMM doesn't support nested virtualization either. I don't know about NetBSD's hypervisor or DragonFly's.

I don't know of any. I don't think the built in OS X one does either.

There is nothing per se hard about implementing it, the problem comes from implementing it in a usable state. Current Virtualization tech uses hardware specific features and instructions to essentially get rid of much of the overhead of translating memory addresses from the guest VM to the host VM. This has to happen at the hardware level because it affects all the built-in caching AMD and Intel bake into their CPUs to make them faster, things like branch prediction and others. This increases die space, the amount of cache space for VMM and many other things.

There is no hardware acceleration for a nested VM. So whereas Qemu has a code path for straight up emulating x86-64 CPUs and can rely on that when nesting, the other hypervisors would need to create their implementation that serves the same function. Bhyve has decided that isn't something they will support.