Lutris 0.5.15 Fixes Crashes When Using Wayland With High DPI Gaming Mice

[oiaohm]

So how much time do you think it takes to send a ping event over a pipe?

Enough time that a game can successfully not receive input to generate a frame perfect that it otherwise would have.

This is the problem minuscule alterations in latency do come noticeable to those doing speed runs and other things.

There is a catch here. Yes you would think the time stamp on input events will fix things. Problem lots of game engines and some common 3d sculpting applications work from the applications own internal time clock so complete disregarding input event timestamps. So speed from input device to application comes critical and preferable constant.

Do note you said if the ping as not been responded to due to some delay stop sending messages. So timing the ping is not your only factor here.

Think if you are running right on edge with time slices that the event buffer is almost full before before you application gets to process and catch up. The ping without responded pong equals that you have stop filling the buffer while it still has space. All those delayed messages. So the effect of the ping/pong to latency is worse than just the cost to send the ping.

Yes the fact you said how long to takes to send a ping as that justifed it says you have not been testing this to edge. Testing to edge you do need software uinput device todo it dependable.

Yes how fast can I push this tests with how many events per second being processed. I did code up a horrible test program. Check the event buffer if not 90% full it would sleep as in scheduler as in use wait(). Only when 90% full would it process the event buffer with all the event cleared. A program stalls out under the gnome ping/pong solution.

Then you do a test case where you can add custom stall amounts. Then you start find when you start hitting the failure in pong in time input latency goes up with gnome at that point.

Mostly from what I have seen indepe you have been testing the case you are debugging the application and your bring the application to complete stop then run out of buffer. You were not testing how it responds if you would be getting close to buffer full but not quite making it to buffer full.

Yes there are games that have a rendering loop and event loop that work that when they are in the event loop they process all events that are in buffer then switch to the rendering loop for X amount of time then switch back to the event loop. Different class of badly coded. Yes some of those are really bad if the event buffer was made larger as well because the game is going to start graphical stuttering because it spending too long processing events.

No, the problem is that in spite of the knowledge you get from reading bug reports and other sources, you know too much about problems (which you call "dragons") and too little about solutions.

I also know more end users than you.

I am admit we have 2 groups of users here. In fact we have 3.
1) General users like you. You want applications not be distrupted and are willing to take latency hit that you call minuscule.
2) speed runs/some of your 3d sculpting where minor alterations in latency are going to get noticed. Yes 1ns and less counts to these users..
3) Embedded critical safety users are going do the ping and failure to pong will be application termination and restart application because the program has have a issue because it not responding so need to be replaced because it not responding.

One of the things about the open source world you can bet when there are 3 groups of uses there will be a customized solutions for all 3 with incompatibilities.

You want to say I know too little about the solutions. Problem is you are missing key big picture bit. You have not looked at what the user groups are.

indepe the 3 groups of users you cannot make all 100 percent happy with the same solution. Gnome route is more the general user route who never going to run anything that will depend on the input latency being insanely low.

The 3 group in this case we can most ignore they are not going to change from application non responding we will kill it.

Thing is I have done broader tests with automation comparing to the different use cases.

[indepe]

Enough time that a game can successfully not receive input to generate a frame perfect that it otherwise would have.

LOL.

Regarding games, btw, you seem to be talking about those that typically run in full screen mode. Those I would probably let by-pass the compositor in many ways and maybe even read libinput directly (although even that wouldn't save much time). And/or a buffer large enough that it gives it enough rope to hang itself.

You seem under the impression that you are giving me things to think about. Not from my point of view. This was mostly me trying to explain solutions, and I believe between this discussion and the previous one I have given the means to solve all issues except the file handles, the solutions to which I won't let you put your spin on.

[oiaohm]

Regarding games, btw, you seem to be talking about those that typically run in full screen mode. Those I would probably let by-pass the compositor in many ways and maybe even read libinput directly (although even that wouldn't save much time). And/or a buffer large enough that it gives it enough rope to hang itself.

This is what happens when someone presumes. Proton/wine games most of them don't in fact bi-pass the compositor for mouse and keyboard. This is common for mouse/keyboard. Game controllers games have habit of going directly.

indepe think alt-tab out of game here. If game is getting all input from keyboard you cannot alt-tab out of course once you tab out you want mouse back right so the compositor has to keep control of keyboard and mouse in most cases..

You seem under the impression that you are giving me things to think about. Not from my point of view. This was mostly me trying to explain solutions, and I believe between this discussion and the previous one I have given the means to solve all issues except the file handles, the solutions to which I won't let you put your spin on.

All except one argument. This is I can cure you by all illness by killing you argument. Problem is that file handle point exceeding 1024 is instant termination. Compositor disconnecting application is not instant termination. Disconnected application can shutdown cleanly if they don't perform re-connection so making user data loss as small as possible.

Buffer large enough to hang self. Disconnection is that application has not be able to cause event that absolutely says death has to happen.

Also you say solves all issues at what price. No such thing as a free lunch.

The gnome solution of stop sending input stalls out complete by the way if you write application that demards the event buffer be 90% full before it processes it. This comes a dead lock. Where gnome has sent the application the ping and the application never send a pong because it does not get enough events to process yes gnome stop sending applications events because it declares the application non responsive Some badly written games are like this and so stall under gnome yet work under KDE and wlroots.

This problem is more complex than it first appears. There are many solutions that people say do this cure.
1) Expand buffers this lead you to running out of file handles and instant death because wayland prtocol is using local socket/file handle combination.
2) detect non responsive and stop sending new events to non responsive that leads to false positives on non responsive that also causes applications to stall out for no good reason because they have not been sent enough events for them to process..
3) disconnect when socket buffer is full. This has lack of tolerance to stalled out applications and applications lacking reconnect code but with close to zero input latency over head and zero false positives it only disconnecting applications that are not keeping up on event processing because there is some issue.

indepe what do you think is the price of the method you are suggesting in input latency and stall risk.

[indepe]

This is what happens when someone presumes. Proton/wine games most of them don't in fact bi-pass the compositor for mouse and keyboard. This is common for mouse/keyboard. Game controllers games have habit of going directly.

Hmm? I didn't presume, I talked about the possibility of "maybe" letting them do that, but added that even that wouldn't save much time.

Anyway I'm out of this discussion.

[oiaohm]

Hmm? I didn't presume, I talked about the possibility of "maybe" letting them do that, but added that even that wouldn't save much time.

indepe think alt-tab out of game here. If game is getting all input from keyboard you cannot alt-tab out of course once you tab out you want mouse back right so the compositor has to keep control of keyboard and mouse in most cases..

This bit I wrote before why this most not a maybe. Yes going directly to HID devices and the like is used for controllers. Keyboard and mouse are special. Games know they could get better performance going direct but going direct with keyboard and mouse break OS integration to the point users will not tolerate.

You were over presuming on what could "by-pass the compositor". Mouse and keyboard like it or not if the application is running not as solo application has to go though compositor/xserver/something to manage them.

Controllers done by /dev/hidraw is because there is enough of a saving to justify going completely around the compositor when there is not a environment integration issue.

The problem here is what you class is wouldn't save much time. The reality here we would have direct keyboard and mouse with games if there was not the OS integration issue. Game developers and players go after these small saving and only don't go after them if there is some other bad side effect form their point of view.

[indepe]

... If game is getting all input from keyboard you cannot alt-tab out of course once you tab out you want mouse back right so the compositor has to keep control of keyboard and mouse in most cases...

And what keeps the game (or SDL) from recognizing alt-tab and then calling the compositor to do whatever it would do if it had recognized alt-tab itself ?

Anyway that's completely beside the point from my perspective and I don't want to get side-tracked by this discussion anymore. Sorry.

[oiaohm]

And what keeps the game (or SDL) from recognizing alt-tab and then calling the compositor to do whatever it would do if it had recognized alt-tab itself ?

Game developers where given the chance todo this under MS Windows yes the result was when game locked up user end up with their computer jammed as well.

The Host desktop monitoring keyboard at least is required as watchdog.

https://techcommunity.microsoft.com/t5/windows-11/freeze-when-alt-tabbing-out-of-games/m-p/3382508

Please note above the idea of trust game developers is not option. We have these copyprotections that developed decided you alt-tab out you have to be cheating is in their ideal world they would love to disable alt-tab and Windows SAK contrl-alt-delete. Linux does not have a non user disruptive SAK.

Yes you have to remember game developers have chosen to make their game crash on alt-tab under Windows.

indepe like it or not you cannot trust applications with full mouse or keyboard control. Microsoft and Apple have had ideas of doing this at different times resulting in stack of different failures on users that were caused by intentional game design.

Yes trust a game to collect and process alt-tab means you need to trust that a game will due to miss guided idea of what stops cheating will process alt-tab and not tell the compositor by design causing user to be locked into the game..

indepe this is once bitten twice shy. Game developers the saying "Give him an inch and he'll take a mile" really applies. OS you give inch on interface control of keyboard and mouse don't be surprised when game developer decides to take all the control and screw everything else.

Remember making a desktop environment on a computer you are trying to make something that responded the same way to the user over and over again. Not have cases that alt-tab works with most applications but then particular ones it don't work at all.

indepe you just over trusted games. There are design limitations here that you cannot break or bad things will happen.

[indepe]

Game developers where given the chance todo this under MS Windows yes the result was when game locked up user end up with their computer jammed as well.

The Host desktop monitoring keyboard at least is required as watchdog.

Right, it would require some way to pull back the mouse/keyboard control in case of unresponsiveness after a few seconds. Maybe the compositor can continue to receive a duplicate of the keyboard events, or just a note if there are special keycodes in it? I don't know the input stack well enough to tell how to do it, but I'd think there should be a way. (In the extreme, with support from the kernel.)

However as I said the possible time savings are small according to my measurements, so I have doubts it would be worth the effort anyway.

[oiaohm]

Right, it would require some way to pull back the mouse/keyboard control in case of unresponsiveness after a few seconds. Maybe the compositor can continue to receive a duplicate of the keyboard events, or just a note if there are special keycodes in it? I don't know the input stack well enough to tell how to do it, but I'd think there should be a way. (In the extreme, with support from the kernel.)

This is what happens when you don't know the problem space.

https://github.com/snyball/hawck

Todo what you are talking about to recieve duplicate input you would have todo what hawck is doing here.

Input devices you run into the same problem what sound servers for audio solved if you wish to push keyboard and mouse outside of Wayland or X11. Where you will need to implement a routing server.

Using usbguard/Linux kernel you can force disconnect game controllers and watch games with direct connection to game controllers straight up die. Same problem with direct to alsa audio programs when you would attempt to force pull back control of the audio.

That is also another problem the Linux kernel idea of hey I trap a key and do something.

https://www.kernel.org/doc/Documentation/SAK.txt

Yes lets just kill everything.

It would be nice of the Linux kernel had a C2 SAK key like windows. Yes Windows SAK key is control-alt-delete. Notice how that takes you back to your Windows log in screen without kill your session. Linux kernel version of SAK is kind mega nasty and BSD version is just as bad because they don't have a SAK at all.

https://en.m.wikipedia.org/wiki/Secure_attention_key

Yes it would be nice if the Linux kernel had a C2 Sak key. Yes C2 Sak key gets to you login screen by suspending the user session so the user can log back into their user session and lose nothing. So this is kernel work required.

Yes the Linux kernel can monitor for trigger keys and does monitor for trigger keys as the SAK code shows but that does not include means from kernel to go hey I got X key transfer control of all input and output to Y application with means to return back to prior applications without the applications knowing a thing as required by a C2 SAK..

However as I said the possible time savings are small according to my measurements, so I have doubts it would be worth the effort anyway.

This is you not knowing enough. Once you have to implement a routing server on evdev to allow applications to get fake so call direct input device so that you can monitor the traffic and take back control in case of non responsive you will in fact end up with more input latency than what Wayland is doing.

Yes having to emulate input device is not for free. Yes i know this from what I do with uinput. Emulating input devices is not exactly cheap. You can measure the price across hawck without you having to code up anything. Remember hawck is only creating 1 fake input device not 1 fake input device per application for every input device the applications are accessing.

Yes the wayland socket buffer limits caused by use of file handles with local sockets and how Linux and bsd local sockets work is a pain in the tail. This pain is nothing compared to letting applications direct libinput access to keyboard and mouse resulting in having to implement pipewire for evdev(linux there is a BSD equal) input.

Basically the kernel feature of C2 SAK is not implemented in BSD or Linux kernels. Input routing as in implementing something like pipewire for keyboard and mouse will add more input latency than having Wayland compositor do what it doing.

indepe with the current Linux/BSD kernel feature sets there is no simple way to fix theses problems. Yes I know it sucks running into I have filled the Wayland socket event buffer so now I cut off this is still way better than having to use Linux kernel SAK key to dig self out of trouble or the fact BSD don't have a SAK key to dig self out of trouble at all.

indepe pulling back control input/output devices under Unix like OS(not unique to Linux/BSD) is nasty because the Unix like OS design has you terminating applications to achieve this. Remember how I said there was a type 3 used in embedded. In embedded you let application for performance use libinput and connect to input devices directly one very big catch when detect non responsive by watchdog you terminate application. None of this simple disconnect the socket or waiting. See the idea of let application directly connect to keyboard and mouse with Linux/bsd/Unix like/Unix kernel design to get back control is kill application. Applications must only be allowed to indirectly connect by some form of proxy if you don't want to be terminating them.

Sorry to say you have been presuming you can do things that you simply cannot do with current Linux and BSD kernels indepe. If course if you want to implement a correct C2 SAK in Linux/BSD then things would change. Yes with some extension on C2 SAK to allow alt-tab and the like to work as well against the applications will that is currently holding the input device. Of course there is still the problem how to transfer the input device without killing the application.

Yes Linux kernel virtual terminals do have way altering input access but they are not setup to have a single shared graphical output and multi virtual terminals for input. If this was supported things would be different.

Basically in theory what you are suggesting indepe could work but we have to work to the limits of the Linux and BSD kernels. and those limitations say no you cannot do that. Kernel changes required to allow applications sharing a single screen to have direct input and to be able to take back control at any time without killing the application is massive number of changes required.

From a security point of view I would love if applications could have direct input with the kernel being able to be the routing server with C2 SAK and a few extras for alt-tab and the like but we don't have this.

[indepe]

This is what happens when you don't know the problem space.

https://github.com/snyball/hawck

Todo what you are talking about to recieve duplicate input you would have todo what hawck is doing here.
...

Maybe that would be somehow convincing of whatever it is supposed to convince me of, if I hadn't already done it myself for the latency test (I was reading the input stream from the kernel at the same time as receiving the same event from the wayland compositor.) It does require root privileges, but I guess the compositor has them.

So you are mostly wasting a lot of time on trying to convince me that something is too difficult, when actually it is something I have already done.

...
Yes the wayland socket buffer limits caused by use of file handles with local sockets and how Linux and bsd local sockets work is a pain in the tail.

There is no such cause-of-pain at all. However as I said I'm not going to discuss this with you, especially with your attitude of dismissing everything I propose.

Sorry to say you have been presuming you can do things that you simply cannot do with current Linux and BSD kernels indepe.

Sorry to say but I wouldn't even consider taking your word for anything like that. And a kernel change doesn't seem out of question either, but I consider it most unlikely to be necessary.

EDIT: After a little more thought, I believe I have a clear-enough picture of how to implement the whole thing. However, I won't have you "review" it.