Linux VKMS Driver Patches Making It Easier To Build Your Own Virtual Graphics Devices

[phoronix]

Phoronix: Linux VKMS Driver Patches Making It Easier To Build Your Own Virtual Graphics Devices

The VKMS driver within the Linux kernel allows for virtual kernel mode-setting as a software-only KMS/DRM driver that can be used for virtual testing purposes and other uses around headless systems. New patches out of Google are working on ConfigFS support for this virtual display driver and from there making it easy to build out your own virtual graphics device configuration with VKMS...

https://www.phoronix.com/news/Linux-VKMS-ConfigFS-Virtual

[tildearrow]

This might allow for things like vncdrm (a virtual DRM/KMS display, complete with framebuffer, X11 and Wayland support and controlled via VNC)...

[Quackdoc]

huh, I was actually looking into doing something like this with vulkan venus. maybe venus can be made to work with this, never heard of vkms till now

[skeevy420]

huh, I was actually looking into doing something like this with vulkan venus. maybe venus can be made to work with this, never heard of vkms till now

I have in the context that it can be used to pirate media.

Phoronix covered VMKS 3 years ago.

[Quackdoc]

I have in the context that it can be used to pirate media.

Phoronix covered VMKS 3 years ago.

my main use for this would be for waydroid, getting the folks who run nvidia cards gpu acceleration, or rather unsupported gpu's at all that support vk. but that would need to be wired up to the venus backend I alluded too

[mdedetrich]

This might allow for things like vncdrm (a virtual DRM/KMS display, complete with framebuffer, X11 and Wayland support and controlled via VNC)...

Is this a better solution to dummy X11 graphics output when doing VNC on headless devices? I also assume this may fix the issue of Wayland not even properly supporting headless for RDP (at least with ubuntu which I am running on some machines).

[oiaohm]

Is this a better solution to dummy X11 graphics output when doing VNC on headless devices? I also assume this may fix the issue of Wayland not even properly supporting headless for RDP (at least with ubuntu which I am running on some machines).

A proper virtual DRM/KMS would allow you to do ctrl-alt-fn between multi X11/Wayland/tty on the headless end. So closer to KVM over IP solution without in fact having a BMC or a KVM over IP box.. Of course something like this will only be possible with VKMS development.

Remember dummy X11 graphics output with VNC if you restart you X11 server that can be you connection lost and you cannot switch to the text mode terminal and back.

[linuxgeex]

A proper virtual DRM/KMS would allow you to do ctrl-alt-fn between multi X11/Wayland/tty on the headless end. So closer to KVM over IP solution without in fact having a BMC or a KVM over IP box.. Of course something like this will only be possible with VKMS development.

Remember dummy X11 graphics output with VNC if you restart you X11 server that can be you connection lost and you cannot switch to the text mode terminal and back.

At least with Xorg and KDM /LightDM you can run X0rfb or x11vnc listening on localhost then connect using SSH port forwarding and a VNC or RDP client. I've even made that work on iOS with 'ish' for the ssh client with port forwarding, and the free Microsoft RFB client. So far I can't find a solution for Wayland. If there was a kernel VNC module that worked with vkms, that would be awesome, as it would sit below the display server layer, and even work with things like ggi, and svgalib or SDL on FB which would be awesome for embedded.

[oiaohm]

At least with Xorg and KDM /LightDM you can run X0rfb or x11vnc listening on localhost then connect using SSH port forwarding and a VNC or RDP client. I've even made that work on iOS with 'ish' for the ssh client with port forwarding, and the free Microsoft RFB client. So far I can't find a solution for Wayland. If there was a kernel VNC module that worked with vkms, that would be awesome, as it would sit below the display server layer, and even work with things like ggi, and svgalib or SDL on FB which would be awesome for embedded.

https://www.youtube.com/watch?v=XkH_jZ21t7g

This is gnome with RDP is covered here with Wayland. This is a case that DM need to catch up. RDP protocol does support sending message to client to reconnect with new credentials to a same location or different location.

Last time I looked at VNC it does not support handover.
Running the X11/wayland compositor as logged in user end up requiring hand over between DM and the user session because they end up running as different users..

Yes X11 protocol does support level of hand over between X11 xservers.

For particular problems vkms VNC/RDP would be really useful.

If you are not needing DM as in using ssh login you can get away with Wayland multi backend where you can start a headless and rdp with RDP on a socket file. Yes pass over RDP socket file over ssh.

https://github.com/isjerryxiao/kmsvnc

This not good quality yet but its enough to demo what can be done. But it does show that VNC/RDP server is possible on AMD and Intel cards going straight to the cards. So vkms that can just create a standard KMS would be enough to let something like kmsvnc to work. Of course Nvidia still had not got house in order providing the standard interfaces from kernel yet.

We are so use to having like x11vnc and X0rfb basically built on top of X11 and not considering if exactly the same thing could be built straight on top of the DRI/KMS layer without any X11/Wayland dependencies.

Of course there are security reasons not to go the direct to KMS interface and do what gnome video example demos.

Yes vkms/KMS based equals as close to a BMC interface as you can get without being a BMC.

Yes software with vkms to create a fake output then software that understands KMS that scrapes the output that kmsvnc demos then run X11/Wayland/TTY on that output then be remote but being treated as if you are local this is something way past your prior solution.

The Gnome RDP based solution with hand off.is more secure than your past solution. I like to see how to setup RDP DM/login solutions more standardized between solutions so you could have the choice different DMs and different environments all providing RDP.

Of course this brings us back to VNC how to fix up its lack of hand over in VNC protocol.

[linuxgeex]

If you are not needing DM as in using ssh login you can get away with Wayland multi backend where you can start a headless and rdp with RDP on a socket file. Yes pass over RDP socket file over ssh.

yeah been using VNC/RDP over autossh for 20+ years, but it doesn't solve the problem of framebuffer/DRM interfaces during initrd before pivot-root.

Yes vkms/KMS based equals as close to a BMC interface as you can get without being a BMC.

Yes software with vkms to create a fake output then software that understands KMS that scrapes the output that kmsvnc demos then run X11/Wayland/TTY on that output then be remote but being treated as if you are local this is something way past your prior solution.

Yes, and the brilliant part of it is that you could have a kernel without any DRM hardware device at all, but still install a new system running a graphical initrd, over the network interface. And that network interface could be Thunderbolt 4, or even ethernet tunneled over SPI, lol.

Of course this brings us back to VNC how to fix up its lack of hand over in VNC protocol.

If you're displaying the virtual DRM device there's no need for handoff, any more than there's need for handoff in the HDMI or DispayPort or VGA protocols. There's no notion of a user at that level. However, the instant you say "network" you do need to consider Access Controls, which is another kettle of fish. VNC's native access control is pretty weak. RDP's full spec is too convoluted for in-kernel, but not for initrd.

My preferences would be that the kernel boots with the linux console on the virtual display device when it detects no display hardware or when it's passed the option at boot. Then the kernel requires a private key from the user which matches a public key on the boot media to establish a TLS connection, and then forward it to whichever back end is ultimately chosen (VNC/RDP/SPICE/QXL/pipewire...) So then the display becomes available once the initrd has brought up the network, and that can happen before LUKS passphrase is prompted for mounting /.

Or, I guess another option would be to tunnel USB over ethernet, and then at the client side, present a USB hub with a virtual display adapter, keyboard, pointer, and storage... That could work with a kernel that has compiled-in support for generic USB storage, display, keyboard and pointer. Then even the initrd could come from the client, lol. AFAIK this is doable with today's mainline kernel.