Wayland 1.20 Planned For Release In December

[oiaohm]

Which compositor ?
There are so many of them.

The major wayland compositor do screen capture using xdg desktop portal

https://github.com/flatpak/xdg-desktop-portal

Yes this is dbus and pipewire combination. That can be done generically as well using direct to DRI. So compositor level and below compositor level are both possible that way.

[billyswong]

No being on purpose does not mean its not a weakness/limitation. This limitation means the more you put in the wayland protocol the more the wayland compositor will have to-do.

Think about IME and System Tray I could keep on listing here a long list of items that people hold up and say Wayland protocol should provide this that are in this camp of these two examples. Then you wake up hang on Wayland protocol has the limitation I list and you say that on purpose. What effect does this have. If IME/System Tray/.. stack of others is part of the wayland protocol you are now demanding that these features be designed into the compositor. Now lets say instead of putting them in wayland protocol we put them as dbus service defined by some standard the change this feature does not have to be implemented in the compositor instead can be implemented as some Wayland client application and mixed and matched with a person compositor choice.

Yes we want to be able to customise our desktop then we want as much as practical done by dbus and other options outside wayland protocol as possible because of this designed in limitation to the wayland protocol.



These design choices are why different ideas for extensions to Wayland protocol are getting rejected.

Yes IME need security to implement on what applications can be IME providers. This is not the only features users ask to be part of Wayland that need security. Screen capture does need security, faking keyboard and mouse does need security..... Remember if you add a heavy security framework to Wayland IPC you will slow down all Wayland IPC operations.

In another post I talked of IME. Actaully I have no opinion on whether IME communication shall be implemented through Wayland or DBus. My only opinion is it is something that should be standardized in display server / compositor. IME need to position its popup surface in touch of the carat of user application text input area. Such positioning / repositioning is the job of some part of the display stack. So the timeline order isn't "implement IME in Wayland protocol -> demand it into compositor". The timeline order is "IME need to be handled by compositor -> let put it in Wayland protocol"

Wayland may want to be a high speed protocol and outsource anything that shall involve permission check, but compositors can't outsource stuff further away.

[oiaohm]

In another post I talked of IME. Actaully I have no opinion on whether IME communication shall be implemented through Wayland or DBus. My only opinion is it is something that should be standardized in display server / compositor. IME need to position its popup surface in touch of the carat of user application text input area. Such positioning / repositioning is the job of some part of the display stack. So the timeline order isn't "implement IME in Wayland protocol -> demand it into compositor". The timeline order is "IME need to be handled by compositor -> let put it in Wayland protocol"

Wayland may want to be a high speed protocol and outsource anything that shall involve permission check, but compositors can't outsource stuff further away.

There is a catch.

https://wiki.archlinux.org/title/fcitx

The reality is IME does not own in the wayland compositor at all. Do note here a Linux IME how you have GTK_IM_MODULE and QT_IM_MODULE. These are toolkit IME. Why has this happen simple where a IME should be placed relative to a text input area so it does not disrupt the user experience need the toolkit involvement. So IME need to be part of the graphical toolkit for the application not part of the compositor to give ideal.

Next if it not part of application toolkit most cases IME better off as the on screen keyboard.

So IME should this be part of display server/compositor the answer is a truthful no. Should IME be part of virtual keyboard software absolutely yes. Should IME be part of your commonly used toolkits again yes.

The reality here billyswong the information to correctly place a IME dialogue on an application the wayland compositor/display server just does not have. Yes that information to place IME window correctly not to over lap with like the Ok/cancel button... is in the application.

Please also remember IME from wayland compositor point of view when you have a IM module in a toolkit the IME window is no different to any other pop dialog from the application.

https://wiki.qt.io/QtCS2021_-_Waylan...le-v4_protocol (yes this is a wayland protocol but could be a dbus)
Please note billyswong you are not alone in the idea that the display server/compositor should do IME. The reality is as people attempt to make the compositor do IME with protocol growing out of control as you need to transfer more and more information from the application to the compositor so that the IME window can be placed correctly. This same problem happened with XIM for X11 result in toolkits for perform making their own IM modules that run in the application to calculate where the IM window should be placed this turned out to be faster because you need todo less transfers this way and also less error prone.

The reality here IME in the wayland display server/compositor is just reinventing the broken implementation of X11 server XIM that applications end up bypassing with toolkit IM modules. Yes total failure to learn from history. Of course history is going to repeat faster.

Long term is a IME protocol in the display server/compositor end up used in great volume the answer is no because its going to be poor performing.

Reality here IME under Windows, Mac OS, iOS and Android is not part of the display server or compositor instead is part of the system wide graphical toolkit that runs in the application process. Its only us in the Linux/Unix world who are stupid enough to still be attempting to put IME in the display server or compositor and then wonder why it does not work everyone else has learnt better by now.

[ehansin]

oiaohm

Thanks for all your input on this thread. I never knew anything about this IPC stuff related to Wayland. I understand enough to get what you are talking about, and this is all very interesting. Thanks!

[billyswong]

"Compositor / Display server in need to coordinate IME" is different from "compositor / display server in need to implement IME themselves". oiaohm tracing from your link there is the https://gitlab.freedesktop.org/wayla...ntation-matrix which you can see that fcitx is also hooking into some not-yet-stablized Wayland protocol.

Also reading from stuff like https://gitlab.freedesktop.org/wayla...d/-/issues/183 shows me Wayland seems designed to refuse clients from setting top level surface position. So from my limited understanding, for the position synchronization to still occur without any extension to Wayland protocol or compositor, either user applications have to own the IME or IMEs have to own the user application. The "other pop dialog from the application" you mentioned are drawn by the applications themselves so they don't touch this ownership / security problem. X11 don't have this get/set surface position restriction, so what is doable in X11 session without X11 coordination may not be doable in Wayland session without Wayland coordination.

In your Windows / Mac / Android scenario, they can let GUI toolkit own both user application and the IME because the correponding API is official there and everyone can rely on it indefinitely. In Linux, I just came across an outdated applications last week. It can no longer run because it was written for Qt3 and modern Linux distributions don't ship that. in Windows, Win32 API will never be out of support. IME is something that can get "outdated" easily and it is irresponsible to put the API hook onto hands of Linux OS component that deprecate / replace itself fast with no fallback plan.

[oiaohm]

"Compositor / Display server in need to coordinate IME" is different from "compositor / display server in need to implement IME themselves". oiaohm tracing from your link there is the https://gitlab.freedesktop.org/wayla...ntation-matrix which you can see that fcitx is also hooking into some not-yet-stablized Wayland protocol.

Also reading from stuff like https://gitlab.freedesktop.org/wayla...d/-/issues/183 shows me Wayland seems designed to refuse clients from setting top level surface position. So from my limited understanding, for the position synchronization to still occur without any extension to Wayland protocol or compositor, either user applications have to own the IME or IMEs have to own the user application. The "other pop dialog from the application" you mentioned are drawn by the applications themselves so they don't touch this ownership / security problem. X11 don't have this get/set surface position restriction, so what is doable in X11 session without X11 coordination may not be doable in Wayland session without Wayland coordination..

This is based on miss understanding. Yes wayland is designed to refuse clients from top level surface position but that not the end of the story.

Here is a good question how many compositors are you allowed under wayland at the same time? The answer is as many as you like because they can be stackable.

Now that Nvidia is moving to GBM/DMABUF things get simpler.

compositor-client is as far as you are thinking. The reality here its valid under wayland todo a compositor-client1-client2. Pop dialog from another application is possible with wayland. compositer then client1 so client1 requests a pop window located at X from compositor now client1 can pass off to client2 dmabuf and input of the pop up window. Yes the way a application could do this is cut down Wayland compositor protocol. Wayland does not say that an application cannot be client and compositor at the same time.

Reality here can do this without ever need to get the exact windows positions yes you can. Does it require looking at this problem and taking it on different yes it does.

Now think about this as a sub client to the application the application toolkit can extend the IME placement rules when ever it likes without having to go back to wayland standard to add in new features to get more information.

Yes we have stacks of protocols attempted to be added to wayland for IME that are just basically lets reivent XIM. Let forget that wayland allows stacking of compositors. So is IME need the wayland protocol itself no. Could IME need a very cut down wayland protocol yes. The way a virtual keyboard gets to feed back keystrokes to the compositor could be used by the IME to feed back keystrokes to the application. The reality is everything to do IME by stacked compositor route is already in the wayland protocol.

[billyswong]

This is based on miss understanding. Yes wayland is designed to refuse clients from top level surface position but that not the end of the story.

Here is a good question how many compositors are you allowed under wayland at the same time? The answer is as many as you like because they can be stackable.

Now that Nvidia is moving to GBM/DMABUF things get simpler.

compositor-client is as far as you are thinking. The reality here its valid under wayland todo a compositor-client1-client2. Pop dialog from another application is possible with wayland. compositer then client1 so client1 requests a pop window located at X from compositor now client1 can pass off to client2 dmabuf and input of the pop up window. Yes the way a application could do this is cut down Wayland compositor protocol. Wayland does not say that an application cannot be client and compositor at the same time.

Reality here can do this without ever need to get the exact windows positions yes you can. Does it require looking at this problem and taking it on different yes it does.

Now think about this as a sub client to the application the application toolkit can extend the IME placement rules when ever it likes without having to go back to wayland standard to add in new features to get more information.

Yes we have stacks of protocols attempted to be added to wayland for IME that are just basically lets reivent XIM. Let forget that wayland allows stacking of compositors. So is IME need the wayland protocol itself no. Could IME need a very cut down wayland protocol yes. The way a virtual keyboard gets to feed back keystrokes to the compositor could be used by the IME to feed back keystrokes to the application. The reality is everything to do IME by stacked compositor route is already in the wayland protocol.

So you are going the "let GUI toolkit own both user application and the IME" route which I also mentioned in my last paragraph in my last post (the part you skipped from quoting). I have discussed why it works for Windows but not so well for Linux there. Here is another discussion on IME problem 10 years ago https://www.ubuntu-tw.org/modules/ne...post_id=195792 (Chinese language post there). The painful experience is, if GUI toolkits don't have a standard protocol above them to communicate with IME (or the GUI toolkits refuse to use that standard protocol) IME support will become and has become a horrible m-over-n combination mess. Every GUI toolkit got tempted to do their own thing and introduce their own bugs.

"... the application toolkit can extend the IME placement rules when ever it likes ..." No, it should be IME to decide where to place the overlay, with applications providing the carat information. Your suggestion shows how East-Asian language users got messed up by GUI toolkits time and again, because they always want to do their own things, disregarding that IME support takes a lot of time and effort and user feedback to get it right. SCIM got trashed because nobody maintain it to catch up with GTK3. I don't see what great innovation happened in text boxes of GTK3 such that we need to reinvent IME protocol for it.

GUI toolkits aren't just GTK and Qt. For IME problem, game engines are their own "GUI toolkits". And then there are also pure OpenGL / Vulkan applications.

[oiaohm]

So you are going the "let GUI toolkit own both user application and the IME" route which I also mentioned in my last paragraph in my last post (the part you skipped from quoting). I have discussed why it works for Windows but not so well for Linux there. Here is another discussion on IME problem 10 years ago https://www.ubuntu-tw.org/modules/ne...post_id=195792 (Chinese language post there). The painful experience is, if GUI toolkits don't have a standard protocol above them to communicate with IME (or the GUI toolkits refuse to use that standard protocol) IME support will become and has become a horrible m-over-n combination mess. Every GUI toolkit got tempted to do their own thing and introduce their own bugs.

No you missed why everyone got tempted to-do their own thing. No you have not really discused why it properly cannot work under Linux. There is a common toolkit libwayland-client.

"... the application toolkit can extend the IME placement rules when ever it likes ..." No, it should be IME to decide where to place the overlay, with applications providing the carat information. Your suggestion shows how East-Asian language users got messed up by GUI toolkits time and again, because they always want to do their own things, disregarding that IME support takes a lot of time and effort and user feedback to get it right. SCIM got trashed because nobody maintain it to catch up with GTK3. I don't see what great innovation happened in text boxes of GTK3 such that we need to reinvent IME protocol for it.

https://gitlab.freedesktop.org/wayla...le-v3.xml#L224
The reality is the IME need lots of information out the application toolkit. So it does not overlap with key controls or duplicates controls controls and so on.

Notice here you putting a ton of stuff that is hard coded into the protocol when you are going to be doing this by compositor. So every time a toolkit added a new input type that takes text the IME need to be extend right.

Reality here is a generic compositor provided IME is going to end up broken. Like do you really want to have to change your complete compositor because the IME needs to be extended because an application you are using has graphical toolkit with a new widget that need a new input define.

IME in the compositor like with X11 server like it or not does not make sense long term. You need to be able to update the IME in alignment with the graphical toolkits.

The reality you have ignored why XIM failed. Yes toolkits doing their own IM also end in failure but for different reasons. Its really simple to ignore why GTK and QT went with their own IME plugin system. Gtk and Qt were not 100 percent right but they were not 100 percent wrong.

billyswong think about it this way. If I suggest putting complete mesa behind wayland IPC you would tell me this was totally stupid but this was done with opengl with X11 at one point. Then I I suggest putting mess in every applicaiton toolkit you would say this was stupid as well. Both are wrong for different reasons. IME is very much the same. IME make no sense at all to be in the compositor. IME makes no sense to be in each graphical library as it own instance. IME would make a lot of sense to be as commonly uses user space library that can be updated without needing to change the display server/compositor.

GUI toolkits aren't just GTK and Qt. For IME problem, game engines are their own "GUI toolkits". And then there are also pure OpenGL / Vulkan applications.

This one is one of the problems you are not in fact considering all applications. What about applications that use a DRM Leases for the screen. A DRM Lease means you cannot put a third party application overlay over the top. This is the same problem IME under windows use to run into with full screen mode until they moved IME into application code libraries. Yes game engines are more likely to use items like windows full screen or something like DRM leases. This is another reason why you need IME in the application side.

Having a IME protocol application side that able to be like it own wayland compositor to the IME would work with DRM Leases.

billyswong is really simple to miss the reason why IME works under windows with full screen applications so well is that its not part compositor/display server. MS Windows absolute positioning has no part in why IME works with windows full screen applications.

Client side IME should not be a after thought.

Billlyswong can gtk/qt work without a wayland compositor or X11 server on Linux? the answer is yes and has been yes for ages. https://gitlab.gnome.org/GNOME/gtk/t...4/gdk/directfb this is the reality.why 10+ years ago fragmentation absolutely was going to happen with IME being in X11 server because gtk and qt running without a wayland compositor or X11 server still need IME solution right. Has this problem gone away no it has not.

Its really simple to overlook that graphical toolkits be it GTK, Qt, Game engine, opengl/vulkan applications.... in all of these software will need IME for cases where X11 server and Wayland compositors don't exist on Linux. So yes like it or not IME has to be some form of application side plugin/library that is standard other wise we will just go around the same loop again.

The basic loops.

1) We we make it generic add it to server/compositor .
2a) Generic features provided by server does not work for our application toolkit so we will make our own client side solution because if we change the server we will break the other applications.
2b) We need our application toolkit to work without server/compositor so we need to implement IME client side and our code base. Since we implemented this client side already we might as well make the server/compositor IME optional plugin so developers can test how application will be with the server/compositor missing.
3) end user end up with a fragmented mess.

We really need something like "libglvnd: the GL Vendor-Neutral Dispatch library" for IME. The reality here you don't like the fragmentation that X11 IME is yet you are doing the same things that long term caused the fragmentation.

Yes the explode of different wayland IME draft standards when you look closer you will see more and more details coming from toolkits based on toolkit widgets that are forever changing things. IME is never going to be a finished thing. So with IME how are you going to update the interface without bringing the house down all the time.

The reality IME just does not fit in the Wayland compositors and X11 servers as IME need to update as often as the applications toolkits do. Yes the fragmented wayland drafts are a direct cause of the miss match. XIM end up not being able to provide enough functionality is also cause of the miss match due to how slowly X11 server updates it protocol compared to graphical toolkits.

[billyswong]

I guess we got some miscommunication oiaohm. But anyway thanks for clearing things up. I welcome anyone who can make something like "libglvnd: the GL Vendor-Neutral Dispatch library" for IME into reality, sitting somewhere more universal than Wayland / X11 / directfb and everyone plug into it. Currently ibus and fcitx can't coexist and run in a computer at the same time.

[oiaohm]

I guess we got some miscommunication oiaohm. But anyway thanks for clearing things up. I welcome anyone who can make something like "libglvnd: the GL Vendor-Neutral Dispatch library" for IME into reality, sitting somewhere more universal than Wayland / X11 / directfb and everyone plug into it. Currently ibus and fcitx can't coexist and run in a computer at the same time.

Problem here we are not going to get something like libglvnd for IME if people with the skills to code this don't wakes up that in the compositor and in the X11 server for IME basically a no go.

We really do need something for IME that does not care if you are using Wayland/X11/something else. The fact we have graphics card vendors now all on the same page on Linux with GBM/DMABUF and input being able to be passed around stable ways makes this very possible. Yes Wayland/X11/directfb/direct kms/dri... going forwards GBM/DMABUF will be common to them all.

Items that should be neutral between what compositor/display server/application doing direct to screen like pop up IME don't own in the wayland protocols.

To be correct IME should be split in two.

IME virtual keyboard. This is a item a user can place where ever on screen does not attempt to integrate with application. As said it just there to let user enter chars and user use brain on what should go where. Yes this will allow users to enter chars into text boxes that are just going to error so this one gets to move not lock step with the toolkit of the application because its the users job to make this work. This one still would be useful to have a protocol for a application plugn for applications that full control the screen.

IME application integrated. This is the one that has the popup window that lines up application text boxes. When users see popup this they expect IME to only provide options to input text compatible with the box. This is horrible one that has to move lock step inline with the toolkit. Need to move lock step inline with the toolkit means this is not something we will want in the wayland protocol because we will not want to have to change the compositor every time IME needs to expand functionality. Yes this need to expand functionality for IME is why XIM under X11 server first falls in a heap. The person complaining about the Linux IME fragmentation by toolkits is the second fall in heap and the person is way too tunnelled visioned to see that IM modules in toolkits is toolkit developers attempting to correct the XIM falls in heap due to not being able to provide what the user expects.

https://gitlab.freedesktop.org/libinput/libei

We are seeing this one being done for virtual input devices. Yes notice how it allows virtual input to be done by uinput and plugged into the Wayland compositor and possible X11 server in future.

So opening up a socket between application and IME is possible over that socket you can pass file handles so that is DMABUF for the buffer for output.

Thing to remember for a long time we did not have genetic graphics GBM/DMABUF stuff. Nvidia is only now just coming around that we need a generic graphical buffer solution that can be passed around not vendor particular crud. This lack of generic use to force going though the X11 server or wayland server or toolkit custom buffer formats. This change is one of the reason why its possible now to look at IME and do it complete different so that IME is its own thing.

Fixing application integrated IME is not a simple problem due to how many parties that are required to be in agreement. Yes application integrated IME cannot get better while people attempt to fix it are not look back at this history to see what in heck went wrong. User complaints at different time frames tell different parts of the story.

Yes user complains before toolkits did their own IM modules have users complaining about XIM interface being too generic for the box they are entering data into. There is really 3 problems to solve here.

1) Too much fragmentation making IME hard to setup.( that was the post you had about people complaining IME is a mess)
2) How to get all the details IME need for application integrated pop ups from the toolkits so that the pop up functions.(this is the older XIM complaints that lead to toolkit IM modules )
3) Need to be able to update IME without requiring to change core parts like the display server/compositor.(this also comes out of the older XIM complaints leading to IM modules in toolkits because the XIM in the X11 server did not have means to pass the information from toolkit widgets to function correctly and making XIM pass the corrected information would require updating the X11 server)

So parties attempted to fix 2 and 3 and created users suffering from issue 1. Of course attempting to implement IME in the wrong place in the wayland protocol is causing problem 1 to go even more out of control. Remember each wayland protocol document is not designed to be changed regularly or quickly because any change required replacing the wayland compositor and the problem with XIM a change is also replace the complete X11 server so creating same defect.

billyswong I know it sounds hard for me to say that IME does not own in the wayland protocol but when you look complete history covering XIM I truly cannot see how attempting to integrate IME into wayland protocol can give any better result than integrating IME as XIM into X11 protocol of old did. The same forces to fragment that happened to XIM will still exist with a wayland protocol based solution.

The one advantage we have now with GBM/DMABUF finally been universal between all graphics drivers we don't need to use the X11 server/wayland compositor to get a universal graphical buffer format any more so its now truly possible to take on the application integrated IME problem with the X11 server/Wayland compositor completely out the mix so straight between application and IME solution.