VirtIO Native Context Being Worked On For AMD Drivers To Enhance VM Performance

[phoronix]

Phoronix: VirtIO Native Context Being Worked On For AMD Drivers To Enhance VM Performance

As part of an AMD effort to enhance the performance of the AMD Linux graphics drivers when running in a virtualized environment, a set of initial patches are pending for Mesa that implement native context support for VirtIO...

https://www.phoronix.com/news/AMDGPU-Mesa-Native-VirtIO-Ctx

[dreamcat4]

it would be nice to better understand what this means, in which practical usage situations. for example if this is any substitute for gpu passthu, or if a single gpu's resources can or cannot be effectively be split between host and guest machine i.e. not requiring two gpus. this is still a bit unclear (to me)... but perhaps once people can try this out later on. would be nice to find out

[rene]

"to beu sed within"

[Developer12]

The key question is if, like with virgl/venus, a single card can be used with multiple VMs using this infrastructure. That's what AMD and nvidia both charge the big bucks for, since enterprise customers and hyperscalars have need to share few GPUs among many lightweight VMs.

This seems more like a kind of OS-assisted PCIe passthrough, so that you can pass a single GPU through to a single VM without any special hardware support.

[marek]

The key question is if, like with virgl/venus, a single card can be used with multiple VMs using this infrastructure. That's what AMD and nvidia both charge the big bucks for, since enterprise customers and hyperscalars have need to share few GPUs among many lightweight VMs.

This seems more like a kind of OS-assisted PCIe passthrough, so that you can pass a single GPU through to a single VM without any special hardware support.

The short answer is YES. It allows multiple VMs to use the same GPU.

PCIe passthrough is different and already supported, but you can only have 1 VM per GPU. The VirtIO native context allows many VMs per GPU.

[polarathene]

for example if this is any substitute for gpu passthu, or if a single gpu's resources can or cannot be effectively be split between host and guest machine i.e. not requiring two gpus.

I don't think it performs on par with passthrough, sounds like there is some overhead as the driver support still uses VirtIO transport to talk to driver on host from the sounds of it. There was a similar driver done like this (MSM I think?) for Qualcomm which Chromebooks could use IIRC.

The benefit AFAIK is that current driver with virglrenderer in the guest would have extra overhead and support, which would send this to the host driver to process again, but this new native context one for AMD would be like a proxy with a bridge (via VirtIO). The guest would apparently see that AMD driver and gain all the benefits of support like the host driver has (compute, vulkan, HW video decode/encode).

Presently there is some HW video decode (not sure about encode?) with QEMU that could offload onto a host driver through an API, but the format support was limited rather than what the host GPU + driver was capable of.

Venus support is also something we've been lacking due to some missing pieces, while ChromeOS has been able to leverage that. With this new AMD driver support you'd have access to Vulkan without the Venus layer AFAIK which is great

---

I don't have an AMD GPU yet, but I have heard the virglrenderer / VirtIO-GPU that we currently have still works pretty well for OpenGL on AMD hardware. I have an Nvidia GTX 1070 which does not though, when I benchmarked it there was huge amounts of overhead in PCIe bandwidth used and 100% GPU utilitization (the same workload on the host was around 30% with minimal PCIe bandwidth), and I recall performance in the guest was extremely poor (plus no CUDA support this way).

VMWare on the other hand retained much more performance that it was acceptable for a guest for me. They would accelerate the guest GPU via some vulkan layer that would run on the host GPU. I was hoping Venus support would land in QEMU to take that advantage away, but I haven't heard of much progress being made since the last Collabora blog post I read on Venus in late 2021.

So this news is just icing on the cake for my next GPU to be AMD

---

SR-IOV is the commercial offering (at least with AMD) for splitting a GPUs resources to multiple VMs, but that'd work with other OS guests too AFAIK.

It'd be fantastic if this new feature would be compatible (or minimal work) to support other guest OS like Windows with AMD drivers. IIRC one of the issues for Windows support with virgl/venus was drivers needing to be signed or something, which shouldn't be an issue for AMD drivers

[Quackdoc]

Nice! this is massive. currently we are stuck with buggy venus and slow virgl. this will be great for MANY people. I can already tell you I plan on testing and using this with android if possible. I have doubt though since android doesn't AX86 doesnt implement any of the split kms driver stuff

I don't think it performs on par with passthrough, sounds like there is some overhead as the driver support still uses VirtIO transport to talk to driver on host from the sounds of it. There was a similar driver done like this (MSM I think?) for Qualcomm which Chromebooks could use IIRC.

It should be pretty close to native perf

The benefit AFAIK is that current driver with virglrenderer in the guest would have extra overhead and support, which would send this to the host driver to process again, but this new native context one for AMD would be like a proxy with a bridge (via VirtIO). The guest would apparently see that AMD driver and gain all the benefits of support like the host driver has (compute, vulkan, HW video decode/encode).

Presently there is some HW video decode (not sure about encode?) with QEMU that could offload onto a host driver through an API, but the format support was limited rather than what the host GPU + driver was capable of.

virgl renderer + qemu does support vaapi but I have yet to get it to work

Venus support is also something we've been lacking due to some missing pieces, while ChromeOS has been able to leverage that. With this new AMD driver support you'd have access to Vulkan without the Venus layer AFAIK which is great

---

I don't have an AMD GPU yet, but I have heard the virglrenderer / VirtIO-GPU that we currently have still works pretty well for OpenGL on AMD hardware. I have an Nvidia GTX 1070 which does not though, when I benchmarked it there was huge amounts of overhead in PCIe bandwidth used and 100% GPU utilitization (the same workload on the host was around 30% with minimal PCIe bandwidth), and I recall performance in the guest was extremely poor (plus no CUDA support this way).

Virgl opengl is bad on all hardware, not just Nvidia, probably because it actually more or less emulated a gallium driver using opengl

VMWare on the other hand retained much more performance that it was acceptable for a guest for me. They would accelerate the guest GPU via some vulkan layer that would run on the host GPU. I was hoping Venus support would land in QEMU to take that advantage away, but I haven't heard of much progress being made since the last Collabora blog post I read on Venus in late 2021.

So this news is just icing on the cake for my next GPU to be AMD

---

SR-IOV is the commercial offering (at least with AMD) for splitting a GPUs resources to multiple VMs, but that'd work with other OS guests too AFAIK.

It'd be fantastic if this new feature would be compatible (or minimal work) to support other guest OS like Windows with AMD drivers. IIRC one of the issues for Windows support with virgl/venus was drivers needing to be signed or something, which shouldn't be an issue for AMD drivers

signing isn't really an issue. if it was rhel drivers would support it. big issue is that
A) Virgl opengl sucks
B) Venus isnt in stable qemu yet (it will be getting picked up again soon according to ML and collabora blog post) so nothing there to work with but crosvm which is kinda abig pain right now

[Quackdoc]

The short answer is YES. It allows multiple VMs to use the same GPU.

PCIe passthrough is different and already supported, but you can only have 1 VM per GPU. The VirtIO native context allows many VMs per GPU.

does AMD support... what was it? kmsro? I cant remeber, but right now an potential issue I see with this is the copy from AMDGPU -> virtio-gpu. and the only mitigation for this I can think of is wayland passthrough. currently for that there is the crosvm solution which uses virtio-gpu, and waypipe with uses sockets but no support for AF_VSOCK yet.

there are other options, but low latency and low overhead (as much as possible ofc) with virtio-gpu would be nice for apus. my experients in the past have been... fine, but not great.

also we might need it for android x86

[edgmnt]

Why did it take so long to virtualize GPUs at this level, especially given that we've had open source drivers for quite some time? It's not like we have to translate Direct3D/OpenGL to OpenGL anymore.

Compared to SR-IOV and passthrough with a good IOMMU, I suppose it lacks some security guarantees, but no less than what you typically get when running multiple applications on the same machine. And performance-wise I guess it's just a layer of indirection with minimal involvement. But the upside is it's much more flexible: no virtual functions, no wholesale passthrough...

I'd still like a bit more isolation, something that could allow one to run fully untrusted VMs and not worry about the card/driver messing stuff up that easily. I suppose that can't be achieved without hardware cooperation (how would the IOMMU distinguish flows meant for different VMs/apps?). But this is already better than ACS overrides since you no longer rely on the guest OS playing nice.

[oiaohm]

Why did it take so long to virtualize GPUs at this level, especially given that we've had open source drivers for quite some time? It's not like we have to translate Direct3D/OpenGL to OpenGL anymore.

Duplicate of effort. MxGPU provided more secure version on Professional and server hardware from AMD. Intel GVT-g that is intel equal to MxGPU is all intel hardware since 2014. This VirtIO Native Context will basically be for the AMD consumer cards and Integrated GPU because the MxGPU will be less overhead on your Professional and server hardware but of course MxGPU. Yes the Mediated Passthrough hardware is in the Professional ans Server AMD cards not in Consumer or Integrated GPUs..

Question of course remains is will AMD drivers for Windows support VirtIO Native Context in time.

VirtIO Native Context by AMD is closer to Nvidia vGPU stuff. Remember Nvidia vGPU is restricted to professional and server cards as well.

edgmnt you also have to remember the core developers making the AMD drivers for Linux are almost all AMD staff so there was a money reason not to possible disrupt Professional/server card sales by making competing solution.

This VirtIO Native Context was developed by a developer working on opensource qualcomm GPU driver freedreno for chromeos funded by google. Qualcomm soc hardware does not have any hardware for Mediated Passthrough like MxGPU or GVT-g and they were wanting to have close to same levels of performance.