A Remote KMS Linux Backend Is Being Worked On That Could Work With VNC

[phoronix]

Phoronix: A Remote KMS Linux Backend Is Being Worked On That Could Work With VNC

Thomas Hellstrom of VMware who has worked on Mesa going back to the Tungsten Graphics days is developing a remote KMS back-end that could be transmitted over VNC or similar protocols...

http://www.phoronix.com/scan.php?page=news_item&px=Hellstrom-Remote-KMS

[boxie]

That's quite a nifty idea, how would that fit in with an accelerated desktop? could you play a game over it?

[emblemparade]

This is not a bad idea and can definitely be useful in some specific situations.

However, it will clearly not provide a good general-purpose remote desktop experience. KMS has no knowledge of things like "windows", "regions", or "surfaces" and can't optimize accordingly. It could only ever do full-screen remoting, which will be bandwidth hungry. It would not be able to compete with the remote desktop experience you get with RDP and RemoteFX. For that, we still want native Wayland remoting.

Also, it's entirely unclear how remoting KMS could be integrated remoting inputs: keyboard and mouse.

But, I still see value. Not every system uses Wayland or even X11, so this could be a good solution for some exotic Linux boxes. And remote virtual terminal gives especially powerful remote control of a system, before it even boots to a desktop, so it could be great for helpdesks trying to remotely solve especially sticky problems (failed desktops).

To the person asking about remoting games: there's probably no benefit to this method over Steam In-Home Streaming, Moonlight, etc. Your main bottleneck for full-screen remote streaming is network bandwidth. Grabbing the screen in the kernel instead of usermode might save you a few CPU cycles, but it won't feel any different.

[RussianNeuroMancer]

Isn't he could just use evdi for this?

https://github.com/DisplayLink/evdi

[Djhg2000]

For games I think it would make more sense to integrate this with LookingGlass https://github.com/gnif/LookingGlass . Actually, for local VMs in general it's probably a better choice than VNC.

But I think VNC makes a lot of sense for something like a large internal server room. With a bit of work you wouldn't need to differentiate between a virtual and a physical machine and basically have a unified management interface for both.

[oiaohm]

However, it will clearly not provide a good general-purpose remote desktop experience. KMS has no knowledge of things like "windows", "regions", or "surfaces" and can't optimize accordingly. It could only ever do full-screen remoting, which will be bandwidth hungry. It would not be able to compete with the remote desktop experience you get with RDP and RemoteFX. For that, we still want native Wayland remoting.

https://en.wikipedia.org/wiki/Teradici#PCoIP_Protocol

I am sorry that you claim that not good general-purpose remote desktop does not really stack-up. PCoIP solutions don't optimise with windows, regions or surfaces and works quite well with just the graphics output buffer. Working with virtual machines getting that extra information is not always possible.

PCoIP does not really show worse bandwidth usage than RDP with RemoteFX.

Also if knowledge of windows/regions/surfaces are important this information does mean have to go to native X11 or Wayland remoting most of this information is in the DRM framebuffer information with wayland. More of windows/regions/surfaces information is kernel mode DRM framebuffer information with X11 as well. Remember under Wayland each window has it own buffer in DRM in kernel mode. To do this to an application you would still need to maintain a full KMS output buffer for the case application gets permission to screen shot the complete screen or the like.

Remember Wayland compositors that are not build for remote what to find KMS output before they work. So first step is implement KMS. Possible latter if there are perform to be gained knowing those extra things someone could extend from KMS remote to a remote that use KMS and DRM information in kernel and this would still stay X11 and Wayland netural.

Also, it's entirely unclear how remoting KMS could be integrated remoting inputs: keyboard and mouse.

Remoting the inputs is insanely easy. With the mouse you could just pretend remote the screen is DSI touch display so able to pass in direct clicks on the image displayed. Keyboard could also be just as simple expand Consumer Electronics Control (CEC) from hdmi. Both of these means all the input you need could come from the display interfaces. Yes the advantage here is the connect and disconnect could be made unified and behave like normal real world hardware.

But, I still see value. Not every system uses Wayland or even X11, so this could be a good solution for some exotic Linux boxes. And remote virtual terminal gives especially powerful remote control of a system, before it even boots to a desktop, so it could be great for helpdesks trying to remotely solve especially sticky problems (failed desktops).

That is the big advantage as it does not care what you are running. Do note above after you have KMS working there is possibilities of extending into general DRM information from kernel and possible providing custom accelerated playback and the like.

To the person asking about remoting games: there's probably no benefit to this method over Steam In-Home Streaming, Moonlight, etc. Your main bottleneck for full-screen remote streaming is network bandwidth. Grabbing the screen in the kernel instead of usermode might save you a few CPU cycles, but it won't feel any different.

Moonlight is only a client NVIDIA GameStream is the protocol and the server side does not work on Linux. So this could allow game streaming that does not depend on having a particular vendor card or particular vendor game supplier. There are a few other things can be done kernel side to effect cpu/gpu schedulers allocated times. With work kernel mode solution should be able to have a little more predictable behaviour. I do agree unlikely to notice difference in most cases.

[boxie]

This is not a bad idea and can definitely be useful in some specific situations.

However, it will clearly not provide a good general-purpose remote desktop experience. KMS has no knowledge of things like "windows", "regions", or "surfaces" and can't optimize accordingly. It could only ever do full-screen remoting, which will be bandwidth hungry. It would not be able to compete with the remote desktop experience you get with RDP and RemoteFX. For that, we still want native Wayland remoting.

Also, it's entirely unclear how remoting KMS could be integrated remoting inputs: keyboard and mouse.

But, I still see value. Not every system uses Wayland or even X11, so this could be a good solution for some exotic Linux boxes. And remote virtual terminal gives especially powerful remote control of a system, before it even boots to a desktop, so it could be great for helpdesks trying to remotely solve especially sticky problems (failed desktops).

To the person asking about remoting games: there's probably no benefit to this method over Steam In-Home Streaming, Moonlight, etc. Your main bottleneck for full-screen remote streaming is network bandwidth. Grabbing the screen in the kernel instead of usermode might save you a few CPU cycles, but it won't feel any different.

I went and read the post and
1) Damage Detection was mentioned (this is how vnc does its updates)
2) injecting remove input events into evdev could be a thing
3) they are thinking of opt in from the hardware drivers, so no gaming unless you have opt in
4) you can then use it also as a way to test different screens (e.g. 4k HiDPI) that you do not have

[thellstrom]

A couple of comments on this:

Input (mouse and keyboard) could be trivially handled by the linux "uinput" functionality. Compositors and X servers immediately recognize the new input devices.

Damage will be primarily handled through a new KMS damage interface. Google for "page flip with damage".
In case damage is not available, For contents readily available to the CPU, it's currently much cheaper to compute the diffing region before encoding than to try to encode the whole framebuffer.
For contents residing on the GPU one needs to resort to other techniques, one of which could be hardware encoders and a solution needs to be able to provide for such techniques to be used.

Gaming is not a primary use-case and as previously mentioned, network bandwidth and stack latency would be the limiting factor for gaming on this solution.

[DrYak]

That's quite a nifty idea, how would that fit in with an accelerated desktop? could you play a game over it?

I think that remote gaming would definitely be an interesting application.

However, it will clearly not provide a good general-purpose remote desktop experience. KMS has no knowledge of things like "windows", "regions", or "surfaces" and can't optimize accordingly.

...because for games (specially the 3D-intensive type like first person shooter that one would run remotely instead of directly on the tablet/smartphone/whatever) tend to not benefit much from "damage".
Bascially the whole screen is changing, not special small subrect in some app's window (unlike desktop application, or casual gaming app that are less screen intensive - thing some 2D puzzle game)

(Also note that such a system could optionnally be provided "damage" informations)

It could only ever do full-screen remoting, which will be bandwidth hungry.

Huh, nope. The whole purpose of any half-decent remoting protocol is to do decent compression to reduce bandwidth consumption.
The only difference is the specific strategy you're using.
- VNC and co will mostly rely on damage to know which sub part of the screen are changing, and only transmit these rectangle (and do some lossless or lossy compresison on the subrectangles).
- for full screen remoting (like graphics intensive games), you basically compress the screen as a video stream.

To the person asking about remoting games: there's probably no benefit to this method over Steam In-Home Streaming, Moonlight, etc. Your main bottleneck for full-screen remote streaming is network bandwidth. Grabbing the screen in the kernel instead of usermode might save you a few CPU cycles, but it won't feel any different.

Actually, it's *extremely useful* because most of the modern graphic cards have video compression hardware.
By being in-kernel, this remoting service could very easily do the compression using hardware and without copying buffers around.

Basically by combining this current idea (have the remoting server run as an extra display target in the KMS stack) and the idea to be able to use Video4Linux as display target (was mentionned some times ago as a way to do screen grabbing) and the hardware compressiong, and also the readily available networking drivers,
you could design a *very low latency* game streaming feature, simply by stringing together drivers, without ever needing to switch to user-space context nor needing to copy buffers around.

(It's a bit reminiscent of the high speeds/low latencies that the in-kernel nfs server is achieving by stringing together various functionality - such as filesystem drivers, network drivers, etc. - already present in the kernel)

[boxie]

- for full screen remoting (like graphics intensive games), you basically compress the screen as a video stream.

and look - we have a shiny new AV1 codec that'd be perfect for it