Lutris 0.5.15 Fixes Crashes When Using Wayland With High DPI Gaming Mice

[indepe]

Read the like this time. GDM/SDDM have nothing todo with Gnome/Mutter and Plasma/KWin other than starting them.

The reason I mention GDM/SDDM is not because I would think they are already being used for that purpose, but because they could be. They have root access and they exists from the beginning of the Wayland environment coming up. Getting evdev access is not difficult. The questions is more to know when to cutoff access. However that question would appear to have multiple answers.

The two texts are apparently both very old, the first one talks about starting Weston from a command line terminal, and things like that. So unrelated to today's common situation. The second text is from 2014 and has remarks that would appear to only support the possibility of what I am talking about, regarding libinput and such.

However I am not interested in actually writing the code for you, and figuring out where exactly that code would best reside. I just know from my own tests that the Linux kernel provides everything needed, and it would be up to the infrastructure above to determine the best plumbing. It is not in the specification of some existing process? Who cares, of course not, that feature doesn't exist yet, above the kernel. My point is, the kernel would support it, and it can be done.

Think person on a laptop playing a game and recording with OBS with only a single screen. Speed of swapping between full screen game and OBS does come important particular with games that don't pause when put in background.

Sorry that 10ms to 100ms or more does matter. The switching need to be fast from full screen to another application and back for quite a few use cases indepe.

I had to literally lough out loud about this. It wouldn't even take close 100 ms, more likely significantly less than 10 ms, but this is nonsense anyway. 10 ms wouldn't even be noticeable in a situation where you need a keystroke to go in and out of full screen mode. There is no need to support someone speed-flipping a full-screen game to other apps and back, in less than 1/10 of a second. So many things have to happen during such a double switch. Have you measured how long that takes with the current Wayland compositors?

Actually I'm getting the impression that you seem completely uninterested in the possibility of a solution to reduce the latency of the game itself. "Too much work"? Right...

Also what about the case of nested compositors Yes nested compositor can be full screen with application inside it full screen..

Again I doubt such a setup would have low latency in the first place.

[oiaohm]

The reason I mention GDM/SDDM is not because I would think they are already being used for that purpose,

https://consolekit2.github.io/ConsoleKit2/#idm139836753009248

The reality is GDM/SDDM and all other display managers have been off problem. Before logind/seatd you have consolekit doing the job.

indepe there has been a display manger and a seat manager for a very long time. The one in charge of who gets access to the input devices and output devices is the seat manager. consolekit is the old seqt manager with consolekit2 being the fork for those who did not want to go systemd or for BSD users that don't want to use seatd.

The two texts are apparently both very old, the first one talks about starting Weston from a command line terminal, and things like that. So unrelated to today's common situation. The second text is from 2014 and has remarks that would appear to only support the possibility of what I am talking about, regarding libinput and such.

Hard fact when you start a Desktop envronment from display manager its running on a hidden terminal.
/usr/share/xsessions and /usr/share/wayland-sessions look in side the .desktop files here.

Yes you start a KDE X11 session its exec /usr/bin/startplasma-x11 started in hidden terminal by the display manager as your user. The Wayland version is "/usr/lib/x86_64-linux-gnu/libexec/plasma-dbus-run-session-if-needed /usr/bin/startplasma-wayland" This is under debian of course.

The reality here is a display manger does the follow.
1) get the username/password/... as in the information pam(and before) wants to confirm that you are a valid user.
2) informs the seat manger what ever it about the user who should be on the seat.
3) setup some things like Xsession key and where X11 clients should be connecting to.
4) Does equal to su –c [command] [other_user] with command being what ever in the .desktop files or historic custom display manager files.

Yes display manager with number 4 it sits there waiting to command to end and when command ends the display manager informs seat manager to give it back the input.

Display manager job all the way back into 1988 when display managers was added to the X11 protocol has been exactly the same thing.

Consolekit was the seat manager of choice from 1994 on fairly much all Unixs, Linux and BSDs until systemd takes off in 2013. There are different older seat managers that even predate display mangers existing but these were very much stack of platform custom scripts..

indepe display manager is one of those really stupid names. Yes X11 protocol calls it a display manager but its never been in charge of display or input devices. X11 server, seat manager and other security parts (like pam) have always been in charge of the display/input device access. Remember no display manger exists before 1988. Yes text based logins before that yes where people would set there X11 environment as there shell.

X11 display managers are just over grown process watchdogs running with high privilege that happens to check you credentials before it runs the process it need to watchdog..

You need to read that 2014 more closely. libinput is using logind to get the device handle.

https://blog.martin-graesslin.com/blog/2016/12/how-input-works-creating-a-device/

But if KWin runs as a normal user and the user is not able to read from the device files, how can KWin read them? For this we need some support. Libinput is prepared for the situation and doesn’t try to open the files itself, but invokes an open_restricted function the library user has to provide. KWin does so and outsources the task to open the file to logind. Logind allows one process to take control over the current session. And this session controller is allowed to open some device files. So KWin interacts with logind’s dbus API to become the session controller and then opens the device files through the logind API and passes them back to libinput.

Here is 2016. Do you need the 2018/2020 and 2021 when he repeats himself. The way this works has basically never changed. Display mangers absolute is not involved directly.

I had to literally lough out loud about this. It wouldn't even take close 100 ms, more likely significantly less than 10 ms, but this is nonsense anyway. 10 ms wouldn't even be noticeable in a situation where you need a keystroke to go in and out of full screen mode. There is no need to support someone speed-flipping a full-screen game to other apps and back, in less than 1/10 of a second. So many things have to happen during such a double switch. Have you measured how long that takes with the current Wayland compositors?

How are you going to give the process that you have killed it input file handle on because it lost focus a new handle? Remember the Wayland compositor it self cannot create a non duplicated handle. The IPC communication to provide a new handle.

Many thing have to happen with a double switch. Can a Wayland compositor cheat the answer is yes when it the input device proxy like it currently is.

Wayland compositor current design can take application off screen put another application on screen and put the application back on screen without sending the application that been displaced off screen any messages being sent the application.

Remember this has all been done with only the wayland compositor having CPU time. Something like EVIOCMUTE would allow applications with direct input access for low latency to be taken off screen without needing that process to have CPU time or waste CPU time recreating buffers.

Really its pointless to measure it on current wayland compositors without understand what you are saying is going to cost.

open_restricted and closed restricted of libinput is not a zero cost thing.

Another thing with using EVIOCREVOKE in the switch away case is that it leaves the file handle standing to be closed. If application fails not notice EVIOCREVOKE has been put in the file handle it using it can sit on a dead file handle basically for ever and the users like why did this application fail to get input.

EVIOCMUTE was designed that the compositor was turn on and off the input to the application without the application having to do anything. This is about reducing the amount of code that has to be correct.

Actually I'm getting the impression that you seem completely uninterested in the possibility of a solution to reduce the latency of the game itself. "Too much work"? Right...

Wrong I have looked problem. Mapped out where the road blocks are.

Road block 1 is that the current used seat managers don't provide support to hand over non duplicated file handles in way that compositor can trust that fact.
Road block 2 the current Linux desktop is going to have too many places of failure. EVIOCREVOKE was not designed for the task.

Creating and destroying input stacks can very quickly eat up all the performance saving you have by allowing application direct input device access. Worse is when application gets out of sync so stop getting input because it fail to notice the handle it has happens to be dead because of EVIOCREVOKE.

Again I doubt such a setup would have low latency in the first place.

One of your nested compositors is gamescope by valve. So law latency for nested compositors is required like it or not.

Due to Wayland protocol using DMABUF so much we end up with GPU syncing. As even the gamescope developers document on the graphical side gamescope nested and not nested there is close to zero performance difference. So close they cannot measure the difference.

The graphical side of Wayland protocol design is insanely low latency since 2013 when DMABUF was introduced to the protocol so allowing GPU accelerated sync.

The only real performance bottleneck in the Wayland protocol is this input stuff. I really do wish in 2013 that the input side has got acceptable solution.

The reality is the idea you are throwing around was debated with EVIOCMUTE. Acceptable non provillaged way of doing EVIOCMUTE that wayland compositors could use for applications never came. EVIOCREVOKE was cut back to something acceptable to plug the security issue this is absolutely not designed for the input performance problem.

The input performance problem is revisit EVIOMUTE and work out acceptable solution.

You need the wayland compositor be able to stop and start input access on applications without needing application direct involvement. We don't have the Linux kernel feature mainline for this. Wayland protocol input handling is a compoised design because they could not get something like EVIOCMUTE mainline.

[indepe]

At this point I think it would be good to reflect on the larger picture, however at the moment I'm too busy to write something. Yet soon. You won't be surprised to hear that it hasn't changed much.

[oiaohm]

At this point I think it would be good to reflect on the larger picture, however at the moment I'm too busy to write something. Yet soon. You won't be surprised to hear that it hasn't changed much.

Thing is I have been looking at the larger picture of this problem. That why there are 2 particular problems with doing direct for keyboard and mouse. Avoiding those problems and doing highly optimized graphics create the file handle issue.

The existing big picture has particular limitations that require core kernel work to fix.

[indepe]

The existing big picture has particular limitations that require core kernel work to fix.

Then how come compositors can work with the existing features? Just need to pass on/duplicate these features to a "guest".

(I will soon be working on latency again, then I will write more.)

[oiaohm]

Then how come compositors can work with the existing features? Just need to pass on/duplicate these features to a "guest".

Wayland original lead developer in 2013 did the EVIOCMUTE work by the way.

https://patchwork.kernel.org/project/linux-input/patch/[email protected]/

Yes this is the same Kristian Høgsberg who wrote

https://wayland.freedesktop.org/docs/html/

Yes same email address and all.

The developer of Wayland protocol look at doing the same thing you are talking about.

Current Wayland protocol has the compositor being a input proxy to the Wayland application.

EVIOCREVOKE

https://patchwork.kernel.org/project/linux-input/patch/[email protected]/

Now this is KMSCOM developer David Herrmann. Yes this makes it into mainline but the feature need to allow direct input to be passed safely over to applications is not include. This is to make sure nothing else is still using the input.

The reality here is the idea of allowing applications to have direct input instead of input being proxyed by the wayland compositor was considered back in 2010 to 2012 given up on when EVIOCMUTE was not merged.

The issue with the event buffer limit from the Wayland compositor to the application caused by input devices would not now exist if the EVIOCMUTE had be merged then the Wayland protocol alterations in 2013 were also merged to used direct device handles. Yes there is extra things Wayland protocol need to add like informing application what the current keyboard layout is so they can process direct input correctly these extra parts of the Wayland protocol was dropped as well.

indepe the idea you have is not new. Kristian Høgsberg for the 2012 merge attempt of EVIOCMUTE into the Linux kernel he coded up a functional example. Also Kristian Høgsberg also documented the 2 limitations I keep on bring up that you keep on wanting to ignore.

1) the Seat manager need to be altered to allow for sure non duplicated file handles.
2) you need kernel alteration to allow input pausing so that applications are not needing to redo there evdev connections over and over agtain.

Please note Kristrian 2013 example if you can find has the seat manager(yes application running with root privilege) muting/allowing the input of applications at the wayland compositors(running as user) request. Yes also has the applications getting their input device straight from the seat manager. This have resulted something very different to the current Wayland protocol.

EVIOCMUTE feature set was design for the case of applications getting direct input with compositor being able to take back control at any time with min disruption to application or compositor altering around control.

Yes also would have reduced the amount of code Wayland compositor has to have.

The reality is the existing Wayland compositors input does have overhead caused by having to proxy input to work around not having EVIOCMUTE this is most hidden by the clock monotonic time stamp on all input device events so applications know when the event happened so can calculate out the lag.

Like it or not EVIOCMUTE was designed exactly for the case where you allow all wayland applications to use direct input instead of having delay in being informed about input by the Wayland compositor. Yes this would have allowed pausing a Wayland compositors CPU usage and have the foreground application proceed as normal.


indepe simple problem how do you pause the input on application that has direct input using using existing kernel features. Remember EVIOCREVOKE is kill the input access not pause it. X11 and Wayland and MS Windows and Mac OS and Android when your application loses focus are in fact pausing input access not terminating it.

[indepe]

Oh wow.

indepe the idea you have is not new.

Of course it is not new, EVIOCREVOKE was created a long time ago for very much the same purpose. I found it because I was looking for such a feature , and it is actually implemented and works.

There are probably reasons that it got accepted and EVIOCMUTE didn't. One advantage per the docu at https://patchwork.kernel.org/[email protected]/ is this:

Compared to the EVIOCMUTE proposal, we can avoid the CAP_SYS_ADMIN
restriction now as there is no way to revive the fd again. Hence, a user
is free to call EVIOCREVOKE themself to kill the fd.

Additionally, this ioctl allows multi-layer access-control (again compared
to EVIOCMUTE which was limited to one layer via CAP_SYS_ADMIN). A middle
layer can simply request a new open-file from the layer above and pass it
to the layer below. Now each layer can call EVIOCREVOKE on the fds to
revoke access for all layers below, at the expense of one fd per layer.

As you can see, the idea to "request a new open-file" is part of the design. Certainly for full screen games/apps that go out of full screen mode and back in, it will not be a performance problem, the new file can be requested already the moment the old one is revoked.

So the request can work the same time the user is doing the following:
1. leave full-screen mode.
2. desktop comes up again.
3. User activates another app and/or window.
4. User does something in the other app or window.
5. User chooses full screen mode again.
6. game or app brings up the full screen window.
7. User moves to enter a new event.

During all that, there is more than enough time to open a new file, and pass it along.

[oiaohm]

So the request can work the same time the user is doing the following:
1. leave full-screen mode.
2. desktop comes up again.
3. User activates another app and/or window.
4. User does something in the other app or window.
5. User chooses full screen mode again.
6. game or app brings up the full screen window.
7. User moves to enter a new event.

Is this the correct workflow when you check out how users in fact use systems? Sorry the answer is no it not. This workflow gives you way more time than you really have.

You presumed return to desktop.

Take user doing OBS and Game as a base example on a single monitor item. Both applications are full screen and you are alt tab between them so you are not in fact returning to desktop an activating another application.

User is flicking over to OBS to see that it still recording and pressing buttons to send to OBS as required then flicking back to game.

See your 7 point flow plan does not hold up to real use cases where you will be going full screen application to full screen application. Now when both of those full screen applications both happen to be using direct input this it what going to happen.

Then remember some games to be really horrible detect they have been pushed out of full screen mode instant terminate. This is where it gets tricky right indepe you cannot inform application that it left full screen mode application has to believe it still in full screen mode even when it not visible this is require so that particular games anti cheat software designs don't terminate program. Yes Linux native and in proton both have this problem when it comes to games.

EVIOCMUTE some thing like it is need for games due to some game makers idea of anti cheat design yes this also does mess up recording the game when you have a single screen under MS Windows so this is not a Linux only problem..

Yes you 7 point did not consider that the application holding full screen will terminate if you tell it that its not full screen.

indepe remember how you said look the big picture and I said I had. Part of this was researching into real user workflows include where the gremlins/problems are that you have not done.

Yes indepe look again and take into account that you are switching between two full screen applications both using direct input that need to happen quickly and at least one of those applications you cannot inform that its full screen/direct input has been cut off or it will crash/terminate due to how the maker of that program has coded it.

[indepe]

Take user doing OBS and Game as a base example on a single monitor item. Both applications are full screen and you are alt tab between them so you are not in fact returning to desktop an activating another application.

Good point but it doesn't actually make a difference. Just requires more experience to tell.

Even if it did, the compositor could keep a spare file handle in reserve, being ready in advance.

Then remember some games to be really horrible detect they have been pushed out of full screen mode instant terminate.

The game or its framework (like SDL or Wine) would have to support this feature anyway, it doesn't come for free. Neither does Wayland support. Just like the game needs Wayland support, it would need support for this feature.

EVIOCMUTE some thing like it is need for games due to some game makers idea of anti cheat design yes this also does mess up recording the game when you have a single screen under MS Windows so this is not a Linux only problem..

If you want to wait for EVIOCMUTE you will be waiting a long time. Really long. And all other possibilities have higher latency (unless you wan t to access USB directly).

So the question is are you actually interested in low latency. I say no.

[oiaohm]

The game or its framework (like SDL or Wine) would have to support this feature anyway, it doesn't come for free. Neither does Wayland support. Just like the game needs Wayland support, it would need support for this feature.

To answer question is is framework or game its a bit both.

Its not frameworks like SDL or Wine. Instead its the anti-cheat frameworks. Every client side anti-cheat framework has a client side lose of focus detection. Its the game developer who sets in the anti cheat options what happens when focus is lost this can be nothing, logging, application termination, application having a intentional causes seg fault/other bug so it does not appear as intentional..

Remember just because a game is running under wine does not mean it not using a Linux native anti cheat system to collect all the input events.

The first anti cheats with lose of focus detection cause bad thing to happen to users appeared in 2002.

This bit of fact means you need a feature to lie to application that it still has focus when you have in fact taken focus completely away and give focus back without the application knowing what has happened..

If you want to wait for EVIOCMUTE you will be waiting a long time. Really long. And all other possibilities have higher latency (unless you wan t to access USB directly).

So the question is are you actually interested in low latency. I say no.

This is the wrong call. I would love lower latency but this cannot come at the price of making user experencie with games where user will want to use low latency of making their life worse. Yes not being able to tab out of game to check on OBS/email... so on because if you do the game will go splat will make users experience worse.

indepe yes the 2010-2012 by the core Wayland developers saw this exact problem.

Simple question do you have a option that allows application to get direct input without the application being able to find out when it has lost focus? Remember if you cannot pull this off many different games with their anti cheat system is going to be mucking up on users because you will be allowing those anti cheat systems by not having this feature to prevent alt tabbing or any other switch away method..

This is because I am looking at the bigger picture and where the pit falls are this pit falls limit what are workable options.