Wine Developers Are Working On A New Linux Kernel Sync API To Succeed ESYNC/FSYNC

[indepe]

Stop right there. As long as functional cmpxchg is available on the architecture. There are quite a few architectures where cmpxchg is there and its not 100% functional.

Next is not all real-time code runtime in fact use pthread implementations on Linux this is due to what is broken.

What does that have to do with the ability to implement any kind of locking on modern x86 CPUs? Which is what you claimed, isn't it? That Windows has some kind of locking where the fastpath cannot be implemented in userspace with atomic operations?

So I have just shown you that PI style locking, which rt-mutex is used for, does have fastpath userspace optimizations on modern x86 CPUs.
If you think there is some other kind of locking, name it !

Why would anyone "stop right there"? Surely the largest part, or at least a very significant part of WINE users, want to run modern Windows applications written and compiled for modern x86 CPUs, on these same CPUs. And that's where you can use a userspace fastpath. And not only there.

So on some peculiar other architectures worst case maybe you have to make more syscalls, I don't know those architectures. However that doesn't change the situation on modern x86. Just encapsulate that in the high-level operations like lock/unlock. Also, some architectures will be able to run an x86 emulator with modern x86 instructions, so they will need some functional equivalent to emulate each atomic CPU instruction, given that these are now used all over the place within applications, including Windows applications.

Show us some application code using Windows API, that you think cannot be optimized in userspace (on modern x86). Otherwise your talk doesn't mean anything.

[Weasel]

Stop right there. As long as functional cmpxchg is available on the architecture. There are quite a few architectures where cmpxchg is there and its not 100% functional.

LMFAO.

[oiaohm]

What does that have to do with the ability to implement any kind of locking on modern x86 CPUs? Which is what you claimed, isn't it? That Windows has some kind of locking where the fastpath cannot be implemented in userspace with atomic operations?

So I have just shown you that PI style locking, which rt-mutex is used for, does have fastpath userspace optimizations on modern x86 CPUs.
If you think there is some other kind of locking, name it !

That is close but its not quite it.

Windows has locking that has permission processing on it. rt-mutex does not have permission processing on it. Do it with atomic operations in userspace comes next to impossible.

WaitForMultipleObjects in windows is event based. Locking based on event system on windows is in fact permission processing as part of getting the lock and being allowed to keep the lock. As in if your process is not the right user/permission you will notified of the event. You fastpath userspace optimizations are not designed for having secure process permissions on the lock if you have the lock or not. Fast path is not designed to deal with the case that applications permission to have a lock has been revoked so the lock has to be taken back by force.

All this event based stuff in Windows does not have any fastpath code because that does not work due to this being a permission and lock thing yes this is a really old locking system you find in many systems before atomic instructions. Under wine this results in horrible performance over head as you are basically having to take out a lock to wineserver to have wineserver process if you should get the lock or not.

indepe like it or not there is a limitation to what you can do with atomic memory instructions for fast path locking Atomic memory instructions for locking were not designed for the usage case where you have to check permissions to decide if you get the lock or not or if change of permission results in lock being revoked from you.

There are a few pre atomic locking designs that cannot be done with atomic memory instructions they way Windows event system is designed it is one of them. Please note windows does not have just one of these cases of items that cannot be fastpath solved.

Also this kind of locking issue historically has turned up on Linux x86. Not single system problem but the cluster system problem issue MOSIX and others run into MOSIX extended the kernel adding extra locking that supported permissions and also had no fastpath.

indepe there is a real lack in the Linux mainline kernel include locking methods for the problem cases where you cannot use fastpath userspace optimizations. Case of this problem is you SOC arm with multi cpu clusters with shared memory but no atomic protections this performance thing that you can gain some ram transfer speed by not processing protection in the MMU instead requiring software coded for the hardware. Other place is in fact is your general computer clustering with x86 you see a lot of userspace cluster software that is really working very hard to implement cluster wide locking with permission processing in userspace runs into the same overhead problems wineserver is running into.

Basically there is a complete class of locking primitive the Linux kernel is missing. Its a really old class of locking primitive that was generally used on hardware Linux never supported. But that really old class of locking has it use cases.

[indepe]

That is close but its not quite it.

Windows has locking that has permission processing on it. rt-mutex does not have permission processing on it. Do it with atomic operations in userspace comes next to impossible.

So the permission system is just on top of it and doesn't change the locking method or the lock type.

The question how that can be implemented is a different one, and not part of the proposal that we are discussing. It probably will simply ignore event permissions. It isn't mentioned as part of the proposed API.

Again, you have to distinguish Windows userspace and Linux userspace. Any difficulties and impossibilities a WIndows application has in Widows userspace, do not apply to Linux userspace implementing the WIndows API and/or kernel call. And you are not providing any specific technical reasons why it wouldn't be possible to *implement* event permissions in userspace. I don't think that would have to do much with atomic operations, it would just be normal code running inside a normal lock. But that is beside the topic of this discussion anyway.

WaitForMultipleObjects in windows is event based. Locking based on event system on windows is in fact permission processing as part of getting the lock and being allowed to keep the lock. As in if your process is not the right user/permission you will notified of the event. You fastpath userspace optimizations are not designed for having secure process permissions on the lock if you have the lock or not. Fast path is not designed to deal with the case that applications permission to have a lock has been revoked so the lock has to be taken back by force.

This argument may apply to Windows userspace, however it doesn't apply to Linux userspace (see my explanation of that distinction in previous posts).

[snip]

indepe there is a real lack in the Linux mainline kernel include locking methods for the problem cases where you cannot use fastpath userspace optimizations. Case of this problem is you SOC arm with multi cpu clusters with shared memory but no atomic protections this performance thing that you can gain some ram transfer speed by not processing protection in the MMU instead requiring software coded for the hardware. Other place is in fact is your general computer clustering with x86 you see a lot of userspace cluster software that is really working very hard to implement cluster wide locking with permission processing in userspace runs into the same overhead problems wineserver is running into.

If you really want WINE to implement these features, and to implement them even on these architectures, then special case them for example in the way described in the Linux documentation that I quoted. However I very much doubt that will ever happen. It sounds like a recipe for disaster.

Basically there is a complete class of locking primitive the Linux kernel is missing. Its a really old class of locking primitive that was generally used on hardware Linux never supported. But that really old class of locking has it use cases.

This doesn't make much sense to me, if it is supposed to relate in any way to our discussion. It sounds like you read something without really understanding the context, and how that context is different to the one in this discussion. And don't believe everything you read on the internet.

[oiaohm]

Again, you have to distinguish Windows userspace and Linux userspace. Any difficulties and impossibilities a WIndows application has in Widows userspace, do not apply to Linux userspace implementing the WIndows API and/or kernel call. And you are not providing any specific technical reasons why it wouldn't be possible to *implement* event permissions in userspace.

3. WHY IT CAN'T BE DONE WITH EXISTING TOOLS

https://lore.kernel.org/lkml/[email protected]/T/#u

This lists what breaks before you implement permissions. NtPulseEvent there is nothing like it in atomic system. Linux userspace implementing Windows API does run into hard wall of stuff you cannot do with locks using memory atomic operations.

The reality is there are a complete type of locking primitive missing. The reality is wine project has tried all different things to make event system of windows work using Linux provided parts so far its never worked right. Please note they drop doing the advanced forms because basic forms cannot be done. Yes Wine tells all programs that its running as administrator with high rights to stomp over everything so can ignore some of the permission stuff.

The reality here indepe you have not attempted to implement Windows Event system on Linux provided parts yet saying it has to be possible. Developers of wine have tried for over 20 years. There is the repeated points were wine developers get so far and then the fact you don't have the permission system and you don't it kernel side processed all the time comes back and cause developer have to comprise. That comprise results in programs not working or behaving strangely.

[indepe]

3. WHY IT CAN'T BE DONE WITH EXISTING TOOLS

https://lore.kernel.org/lkml/[email protected]/T/#u

This lists what breaks before you implement permissions. NtPulseEvent there is nothing like it in atomic system. Linux userspace implementing Windows API does run into hard wall of stuff you cannot do with locks using memory atomic operations.

Permissions are not mentioned under point 3, or in that list. The problems with NtPulseEvent are specific to the eventfd implementation. As I posted above, there should be no problem implementing NtPulseEvent without races. My guess would be that these problems are apparently due to side effects outside the locked region caused by the use of eventfd with split set and reset operations.

Simple access flags (wait/modify/read) are mentioned under point 2, but just describe which handle can be used for which operations. Difficulties implementing that in userspace are perhaps due to the specifics of their eventfd implementation, and perhaps that futex implementation is not different enough. Also this seems to have been solved already on a higher level, since the access flags are apparently not part of the proposed API. Either way, that seems a very easy thing to do in general. I have no idea why anyone would think a syscall would even help with that. Just a lock should be fine.

It seems you are just referring to that email. As you can tell, I'm not taking your word for it, or anyone else's, regarding the claim that things are hard or impossible.

The reality is there are a complete type of locking primitive missing.

What specifically are you referring to? I can't make any sense of this claim.

The reality is wine project has tried all different things to make event system of windows work using Linux provided parts so far its never worked right. Please note they drop doing the advanced forms because basic forms cannot be done. Yes Wine tells all programs that its running as administrator with high rights to stomp over everything so can ignore some of the permission stuff.

The reality here indepe you have not attempted to implement Windows Event system on Linux provided parts yet saying it has to be possible. Developers of wine have tried for over 20 years. There is the repeated points were wine developers get so far and then the fact you don't have the permission system and you don't it kernel side processed all the time comes back and cause developer have to comprise. That comprise results in programs not working or behaving strangely.

You can easily have 20 or 40 years of experience without ever seeing the inside of a lock. There is nothing that would tell me that experts in the implementation of synchronization primitives are involved in this proposal.

[oiaohm]

Permissions are not mentioned under point 3, or in that list. The problems with NtPulseEvent are specific to the eventfd implementation. As I posted above, there should be no problem implementing NtPulseEvent without races.

Except its not that simple pre attomic style locking welcome to fun.

http://www.codewarrior.cn/ntdoc/winnt/ex/NtPulseEvent.htm

This function sets an event object to a Signaled state, attempts to satisfy as many waits as possible, and then resets the state of the event object to Not-Signaled

Read that carefully and notice something. Where does it say no race condition and no starting multi-able of the waiting at the same time. There are reasons why Windows developers say that NTPulseEvent is pure evil. It is really simple with NTPulseEvent to cause a race condition and horrible it part of the design . You CPU has 16 threads able to be used a NTPulseEvent is called and there are 16 items waiting on the event all 16 could now be activated at the same time.

Now this is where wait on multi-able objects come in. So you event is wait on one NTPulseEvent and one Exclusive one at a time event this should only get cpu time is both are meet. So you could have a 16 thread system with 16 waiting on the PulseEvent and only 1 start because wait on multi-able they were all sharing the same Exclusive one at a time event. Of course the 16 events on the PulseEvent que could have not confliting event conditions by the wait on multi-able they all can start at once.

This is a completely different beast to what you are thinking of you are needing to solve more than 1 event state at the same time. You have a completely different type of conflicted lock here.

Fun a item might require two different PulseEvent triggered Events to be active at the same time. The idea of being 100 percent race condition free is a new idea.

[indepe]

Read that carefully and notice something.

Sorry, but I can't understand your description, and the link you provided is not very informative.

Why would it set the event state to 'signaled' if then, in the same operation, it resets it? Unless it calls a lower level function like set-event which makes the wake up calls, and this lower level function has a side-effect outside the locked region. In other words, the whole operation is not an atomic whole anymore, the kind of atomic whole that a lock should provide.

I don't know if you will understand my description any better than I understood yours, however what it means is that I think this is an implementation problem. Which perhaps starts with the idea that PulseEvent has to set the event to signaled first. I think it should just reset, and then satisfy the memorized waiters.

EDIT: And if Windows applications depend on any incorrect behavior, then you just simulate it, causing the same kind of problems. It doesn't have to be better than Windows.

[indepe]

Except its not that simple pre attomic style locking welcome to fun.

Thinking about what I just posted, I am wondering: Is "pre attomic style locking" your code word for bug-ridden locking with inherent race conditions?

So you think atomic locking cannot be used to implement faulty logic?

[Weasel]

indepe Dude I told you, he just spams technobabble every time without understanding what he's reading and just links random stuff. You can keep replying to him of course but keep it in mind, so you don't feel like you've wasted your time. I'm not talking about this thread in particular, he always does it.