Announcement

Collapse
No announcement yet.

Wine Developers Are Working On A New Linux Kernel Sync API To Succeed ESYNC/FSYNC

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • indepe
    replied
    Well, I should add that the current WINE implementation, or one of the existing patches, may have bugs that I don't know about. However that is not what I am discussing.

    Leave a comment:


  • indepe
    replied
    Originally posted by oiaohm View Post

    This does not work in all cases wine has implement version B way in current code and it does not work with particular applications.
    So three weeks later you make another post, and it takes me a month to notice ... anyway :

    Originally posted by oiaohm View Post
    The way pulseevent works changes based on what the event.

    There are two event type classes where the current wine does not work.

    Case 1:
    You do pulseevent and only one of the waiters gets woken up. The wait list does not get cleared and the wake up is not random its qued as in fifo first in first out.
    It seems you are talking about so-called auto-reset events. In the context of our discussion, this is just a detail. In this case, the term "any waiters" simply refers to "the oldest waiter, if any".

    (... If you want a more precise term for that case.)

    Originally posted by oiaohm View Post
    Case 2:
    You do pulseevent and only as many waiters get woken up as CPU threads and if cpu is 8 threads and 4 are still running from the last pulseevent only 4 get woken up.
    I believe you are mistaken if you think this has anything to do with which thread is running on which CPU, as I will try to explain :

    This is a different problem. Obviously, running threads will not be woken. They are already awake. In the absence of an add-event-listener function, this means that if these threads call the wait-function too late, they will miss an event. As far as I understand the Windows API, this can be avoided by using set-event (instead of pulse-event) in combination with auto-reset events. This will wake only one thread per event, but it will do so even if the wait-call comes after the event since the event will remain in the "signalled" state until a thread calls wait on it.

    But if you want many threads that do something for each and every event occurrence, and you want to avoid that they can miss an event (so to speak), then the common solution is to have an API that allows each thread to add a listener to each event. So if you have 3 distinct events and 5 threads that want to do something each time one of these 3 events occurs, then you need 15 listeners. (Unless you don't mind if they occasionally miss an occurrence of an event if they are still running.) This listener will remember if an event occurred while the thread was running (before the wait-call).

    I don't know if the Windows API group that we are discussing allows any direct solution for this case. I think not.

    This may indeed cause problems if the application makes incorrect assumptions about the logic that it needs, vs what is provided by pulseEvent or even set-event.

    This cannot be fixed by WINE, this is a problem inherent in the API. If different timings between WINE and Windows cause problems here, that's kind of something that can't really be controlled, and is certainly not a reason to change the Linux kernel API. (If that seems to help it may as well depend on the weather.)

    By the way, this is not a so-called race condition, even though it may appear that way. You might say it is logical problem, with similar consequences, which will occur if an application tries to use the API for something that it isn't fit to handle.

    Originally posted by oiaohm View Post
    Both case 1 and 2 here the waiters are to think the event has never been set while the ones that have been woken up by pulseevent are to think the event is set. These are not atomic.
    Again, I think you are talking about an incorrect use of the PulseEvent API, one that doesn't agree with its defined logic in the first place.

    Originally posted by oiaohm View Post
    Implementing windows PulseEvent is complex because the one function does not do one thing. Yes it processes events but events is a name for object with multi different properties.


    There is a third case of horrible that you can pulseevent and due to event type the thread that that pulseevent has just triggered to be woken up must think the event is not set if it does not have permission to the event status. Fun right thread woken up because event is trigger but has to believe the event is not triggered ever. Yes there are games out there that depend on this behaviour.
    As I said before, I think it is the return code of the wait-function which the thread must inspect to see if the thread was woken by a signalled event, or by some more exceptional circumstance. After the fact, all event states may have changed in possibly unforeseen ways, one way or the other.
    ---

    In summary, there are problems with the Windows API, per definition, that cannot be solved by WINE as an afterthought. I believe the API does not provide direct solutions for use cases that many will expect it to solve. In consequence, some or even many will run into problems caused by misunderstandings.

    It may be that in some cases WINE triggers latent application bugs, but this should not be used to blame any specific WINE implementation of that API. It may depend on the weather and be the other way around on the next day.

    It is probably good to fully understand the problems and short-comings of the Windows API (and the correct theoretical solutions) in order to decide how to implement it in WINE.

    Leave a comment:


  • oiaohm
    replied
    Originally posted by indepe View Post
    The discussion apparently ended in the middle of the question of race conditions with PulseEvent. (Among other things.) So I want to state my own understanding at this point (which didn't really change), for anyone who might be left wondering, as clearly as I can. (However this will not explain the turns the discussion has taken, or any details that I disagree with.)

    There are two basic ideas, for lack of a better word, to implement PulseEvent:

    Version A:
    A1: Reset event
    A2: Release any waiters

    Version B:
    B1: Set event (or perhaps do this only if there are waiters).
    B2: Release any waiters
    B3: Reset event
    This does not work in all cases wine has implement version B way in current code and it does not work with particular applications.

    The way pulseevent works changes based on what the event.

    There are two event type classes where the current wine does not work.

    Case 1:
    You do pulseevent and only one of the waiters gets woken up. The wait list does not get cleared and the wake up is not random its qued as in fifo first in first out.

    Case 2:
    You do pulseevent and only as many waiters get woken up as CPU threads and if cpu is 8 threads and 4 are still running from the last pulseevent only 4 get woken up.

    The case 1 and 2 types don't release any waiters they are only release a select number of waiters at a time. This is you are manually pulse the event when you want it to happen from user space.

    Both case 1 and 2 here the waiters are to think the event has never been set while the ones that have been woken up by pulseevent are to think the event is set. These are not atomic.

    Implementing windows pulseevent is complex because the one function does not do one thing. Yes it processes events but events is a name for object with multi different properties.


    There is a third case of horrible that you can pulseevent and due to event type the thread that that pulseevent has just triggered to be woken up must think the event is not set if it does not have permission to the event status. Fun right thread woken up because event is trigger but has to believe the event is not triggered ever. Yes there are games out there that depend on this behaviour.

    Leave a comment:


  • indepe
    replied
    The discussion apparently ended in the middle of the question of race conditions with PulseEvent. (Among other things.) So I want to state my own understanding at this point (which didn't really change), for anyone who might be left wondering, as clearly as I can. (However this will not explain the turns the discussion has taken, or any details that I disagree with.)

    There are two basic ideas, for lack of a better word, to implement PulseEvent:

    Version A:
    A1: Reset event
    A2: Release any waiters

    Version B:
    B1: Set event (or perhaps do this only if there are waiters).
    B2: Release any waiters
    B3: Reset event

    In order to work without race conditions, Version B requires a lock around the whole sequence of B1, B2 and B3.
    (Unless maybe there is some fancy trick that I can't think of.)

    And this lock needs to be respected by the other API call implementations.
    As the event can only be accessed through other API calls, Version B will appear to the outside world (that is observing this lock) very much like Version A.

    Version B is probably an attempt to reduce the complexity of PulseEvent to the two simpler operations SetEvent and ResetEvent. In my opinion, that fails because this simplification disregards the conceptual need for a lock around both of them.

    Why do I think the lock is necessary? Because I think that (depending on implementation and circumstances) otherwise a thread might successfully wait on the same event twice, when only the first time should succeed at that point in time. And that is in my opinion not in the spirit of PulseEvent, although some applications might not have any problem with that. Any attempt to fix this without the lock (and/or without understanding what is happening) is likely to run into trouble.

    Other than that, I see no reason why PulseEvent would be inherently racy, although some specific implementations, maybe even Windows itself, may have additional problems internally. However I see no need to imitate any such problems. And likely they would be avoidable in the first place, with sufficient effort. Also, of course, applications might misunderstand PulseEvent's logic, and run into problems because of that.

    Version A is more efficient not only because it avoids the SetEvent in step B1, but also because a larger part of the remaining operations can be optimized to occur outside the locked region. Nevertheless, Version A of PulseEvent is still likely to need a lock as well, with the APIs and functionalities as defined by Windows. (Also, PulseEvent would not be the only reason to use locks.) However locks can be implemented to operate fully in userspace when there is no contention, at least on a number of modern CPU classes and architectures, to express it cautiously.

    That's what I can think of right now. Except to say that the Windows Event APIs as a group require a complexity and inefficiency that in my personal experience is not needed in that form.

    Leave a comment:


  • indepe
    replied
    Originally posted by Weasel View Post
    I'll answer this question since I'm familiar with wine.

    Wine's testsuite generally tests behavior on Windows. In theory, it should run successfully on every Windows version (that they care about). Of course, some are more reliable than others, but you get it.

    Some tests fail on Wine, because it is incomplete or other factors. Those are marked with a "todo_wine" macro so that they show up as such in the test logs when ran under wine (Windows runs are unaffected). In general, the Windows tests must pass successfully. In practice this is not the case, due to many tests being unreliable, and some broken Windows versions.

    If a "todo_wine" marked test succeeds, it's treated as a test failure, because it means it was fixed in Wine and as such the todo_wine must be removed. Regardless, the important thing is for tests to pass on Windows first, and then fix Wine so that they also pass on Wine. This makes it more similar to Windows (bug-for-bug compatibility).

    When I say "unreliable" I don't mean some voodoo or black magic bullshit. It's simply in the nature of the test. For example, tests that rely on race conditions are very unreliable in general, as are tests that deal with hardware stuff (e.g. changing display modes).
    Got it. So the statement
    "The wine testsuite around this function showed the behavour"
    could refer to either or both Windows and Wine.

    Leave a comment:


  • Weasel
    replied
    Originally posted by indepe View Post
    It doesn't really matter for the questions we are discussing, but I am curious: What does the testsuite test? The Windows 10 implementation, or WINE's implementation? The version using evenfd?
    I'll answer this question since I'm familiar with wine.

    Wine's testsuite generally tests behavior on Windows. In theory, it should run successfully on every Windows version (that they care about). Of course, some are more reliable than others, but you get it.

    Some tests fail on Wine, because it is incomplete or other factors. Those are marked with a "todo_wine" macro so that they show up as such in the test logs when ran under wine (Windows runs are unaffected). In general, the Windows tests must pass successfully. In practice this is not the case, due to many tests being unreliable, and some broken Windows versions.

    If a "todo_wine" marked test succeeds, it's treated as a test failure, because it means it was fixed in Wine and as such the todo_wine must be removed. Regardless, the important thing is for tests to pass on Windows first, and then fix Wine so that they also pass on Wine. This makes it more similar to Windows (bug-for-bug compatibility).

    When I say "unreliable" I don't mean some voodoo or black magic bullshit. It's simply in the nature of the test. For example, tests that rely on race conditions are very unreliable in general, as are tests that deal with hardware stuff (e.g. changing display modes).
    Last edited by Weasel; 03 February 2021, 12:33 PM.

    Leave a comment:


  • indepe
    replied
    Originally posted by oiaohm View Post
    When people say windows is a hybrid kernel is true in more than one way. Lot of people think of the hybrid between monolithic and microkernel designs but it does not stop there.

    That the fun part of windows. Sections of Windows are still cooperative multitasking. Sections are preemptive multitasking others are old cooperative multitasking style enhanced with preemptive multitasking.

    Fun part here Windows NT when it starts was meant to be multi platform supporting including platforms that could not perform preemptive multitasking. So really core things like the Event system you find its not designed for pure preemptive multitasking world instead its in fact designed for a cooperative multitasking world because of the idea that could work on any CPU/Hardware type. So on Windows you have these nice hybrid between cooperative multitasking and preemptive multitasking applications in games that depend on both behaviours in different areas of their code base.

    Yes the pre atomic style locks I am talking about you would commonly see in your old cooperative multitasking systems.

    Really fun its not that wine should support pure cooperative multitasking applications because those basically don't exist on modern windows. But has to support a hybrid that is somewhere in the middle between cooperative multitasking and preemptive multitasking. Yes its kind important to be aware what kind of area you are in with windows. The really old stuff Windows NT design are pre atomic locking ideas with cooperative multitasking feature mixed in for good measure.

    Now trying to emulate the old areas of the Windows NT design with pure more modern design parts not going to be fun.

    Windows internally is a hybrid mess this makes emulation in particular areas insanely hard.
    Yeah, the idea that Windows is a hybrid of cooperative and preemptive multitasking is "really fun". How is any cooperative MT logic supposed to work if the thread can get preempted at any time? Just trying of understand what you think, at this point. That would seem to defeat the purpose, wouldn't it?

    Every programmer who has used both cooperative and preemptive multitasking would ask this question, wouldn't they?

    Originally posted by oiaohm View Post
    http://undocumented.ntinternals.net/...ulseEvent.html

    When you read the reverse engined studies on it most of them with different errors.

    Step one set the event to signaled state to release the waiting threads and since they are released when it signeled to them them the event is triggered. Fun here is that releases all or one is not complete. The releases is in fact all, one or as many that can be allocated time slices waiting threads based on event_type nothing like being nice bit I do different things.
    You have just pointed straight at one of the goofs why wine implementation is not working right with some applications because when the thread triggered by event fires up the event is meant to be set not being set causes different applications to fail.
    No, no. That link just repeats what I have seen already. What I am wondering is where you get the idea that threads need to see the event as SET after the wait call returns.

    Previously I was assuming that Windows has ReadEvent because the email proposal has NTSYNC_IOCTL_READ_EVENT. So I was wondering if you think that threads may call ReadEvent afterwards and do the wrong thing if ReadEvent returns the event state non-signaled. But I can't find any reference to anything like ReadEvent in Windows documentations. So, in which way do threads depend on the event being SET afterwards? Do you have an example that's just 5 lines of code or so?

    Originally posted by oiaohm View Post
    Notice the fun here that NTPulseEvent triggers threads in 3 different amounts based on the EVENT_TYPE the event past to it has. This is another trap that can really get developers using this function. Think of Event with max number waiting with event_type that that when you do pulse event all trigger is a nice way to stall out windows. Remember the developer could have only been handling Event types that start only 1 thread from NTPulseEvent so gets really caught out.
    I don't understand a word, so to speak. Which 3 amounts? Did you give me the wrong link reference?

    Originally posted by oiaohm View Post
    Please note this documentation here does not cover the started threads what they look like. The documentation on NTPulseEvent triggered threads them shows the event remaining in signalled state from their prospective even after NTPulseEvent is complete as the Event is cleared for everything else. This area of Windows really does not look like a preemptive multitasking OS or modern Locking. This area really shows NT Windows design age as really really old.
    The documentation that you linked, and the text that you quoted?

    No, it does *not* say that the event remains signaled after NTPulseEvent is complete. On the contrary, it says that the event will get reset right after releasing the waiting threads, by NTPulseEvent itself. Is there some other documentation that you have not linked?

    For ease of reference, I'll repeat your quote here:
    Function sets event to signaled state, releases all (or one - dependly of EVENT_TYPE) waiting threads, and resets event to non-signaled state. If they're no waiting threads, NtPulseEvent just clear event state.

    Leave a comment:


  • oiaohm
    replied
    Originally posted by indepe View Post
    Are you perhaps saying WINE should support applications that were written for cooperative multitasking?

    When people say windows is a hybrid kernel is true in more than one way. Lot of people think of the hybrid between monolithic and microkernel designs but it does not stop there.

    That the fun part of windows. Sections of Windows are still cooperative multitasking. Sections are preemptive multitasking others are old cooperative multitasking style enhanced with preemptive multitasking.


    Fun part here Windows NT when it starts was meant to be multi platform supporting including platforms that could not perform preemptive multitasking. So really core things like the Event system you find its not designed for pure preemptive multitasking world instead its in fact designed for a cooperative multitasking world because of the idea that could work on any CPU/Hardware type. So on Windows you have these nice hybrid between cooperative multitasking and preemptive multitasking applications in games that depend on both behaviours in different areas of their code base.

    Yes the pre atomic style locks I am talking about you would commonly see in your old cooperative multitasking systems.

    Really fun its not that wine should support pure cooperative multitasking applications because those basically don't exist on modern windows. But has to support a hybrid that is somewhere in the middle between cooperative multitasking and preemptive multitasking. Yes its kind important to be aware what kind of area you are in with windows. The really old stuff Windows NT design are pre atomic locking ideas with cooperative multitasking feature mixed in for good measure.

    Now trying to emulate the old areas of the Windows NT design with pure more modern design parts not going to be fun.

    Windows internally is a hybrid mess this makes emulation in particular areas insanely hard.

    Originally posted by indepe View Post
    Which sounds like your 'interpretation' of that point is not shared there either, as there is no mention of setting it to signaled at first. So I wonder: where did you get that idea?
    http://undocumented.ntinternals.net/...ulseEvent.html

    When you read the reverse engined studies on it most of them with different errors.

    Function sets event to signaled state, releases all (or one - dependly of EVENT_TYPE) waiting threads, and resets event to non-signaled state. If they're no waiting threads, NtPulseEvent just clear event state.
    Step one set the event to signaled state to release the waiting threads and since they are released when it signeled to them them the event is triggered. Fun here is that releases all or one is not complete. The releases is in fact all, one or as many that can be allocated time slices waiting threads based on event_type nothing like being nice bit I do different things.
    You have just pointed straight at one of the goofs why wine implementation is not working right with some applications because when the thread triggered by event fires up the event is meant to be set not being set causes different applications to fail.

    Notice the fun here that NTPulseEvent triggers threads in 3 different amounts based on the EVENT_TYPE the event past to it has. This is another trap that can really get developers using this function. Think of Event with max number waiting with event_type that that when you do pulse event all trigger is a nice way to stall out windows. Remember the developer could have only been handling Event types that start only 1 thread from NTPulseEvent so gets really caught out.

    Please note this documentation here does not cover the started threads what they look like. The documentation on NTPulseEvent triggered threads them shows the event remaining in signalled state from their prospective even after NTPulseEvent is complete as the Event is cleared for everything else. This area of Windows really does not look like a preemptive multitasking OS or modern Locking. This area really shows NT Windows design age as really really old.

    Leave a comment:


  • indepe
    replied
    Adding to the post before: I see that in the email proposal for "NtPulseEvent" it also says:
    In both cases the event state is *not* changed to signaled.
    Which sounds like your 'interpretation' of that point is not shared there either, as there is no mention of setting it to signaled at first. So I wonder: where did you get that idea?

    Leave a comment:


  • indepe
    replied
    Are you perhaps saying WINE should support applications that were written for cooperative multitasking?

    Leave a comment:

Working...
X