Announcement

Collapse
No announcement yet.

Windows NT Sync Driver Proposed For The Linux Kernel - Better Wine Performance

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • indepe
    replied
    Originally posted by Weasel View Post
    That's totally fine though? ntsync is called by any thread directly that wants to use it, not the wineserver. It's no different than futex2 or any other sync syscall in the kernel in this respect.
    Yes, I see it as just one of several considerations when comparing alternatives. Personally I like to use multiple (actually many) threads, yet separate them by functionality, so that each functionality can be implemented without using locks. It simplifies the code and it becomes easier to use optimizations. I have encountered situations where it became quite time consuming to avoid internal deadlocks during development, when using locks. However I'd see this as less of a problem for ntsync than for the Wine-server in general, as I would expect it has a large varied interacting complexity. Which is one reason I currently like the idea of a separate event server.

    Leave a comment:


  • Weasel
    replied
    Originally posted by indepe View Post
    Yes, I know, however that is what the discussion was originally about (when you quoted someone else, but accidentally made it look like I wrote that), at least as I understood it. The wine-server can't deadlock itself with internal locks insofar as it isn't using any itself. It just manages locks on behalf of the Windows-apps. So that might cause the Windows-app threads to deadlock, or more likely, they might lock themselves, but not the wine-server thread.

    However ntsync code can theoretically deadlock itself with internal locks (one of them being that "wait-all-lock" which was discussed) because they guard memory that is used by multiple threads inside the Linux kernel ('shared' memory that isn't declared as such because it is used inside the kernel only). When two Windows apps call ntsync functions at the same time, there are two threads inside the Linux kernel using ntsync code. So now the ntsync code could deadlock itself theoretically, using internal locks, if that code uses these internal locks incorrectly.

    Probably that sounds more complicated than it is....

    EDIT: The reason the same functionality can't deadlock by internal locks, is that the Wine-server's interface serializes the functionalty into a single thread, so it doesn't need locks anymore (at the expense of not being able to use more than one core at the same time, for this functionality).
    That's totally fine though? ntsync is called by any thread directly that wants to use it, not the wineserver. It's no different than futex2 or any other sync syscall in the kernel in this respect.

    Leave a comment:


  • indepe
    replied
    Originally posted by Weasel View Post
    It's not about internal locks, it just doesn't account for arbitrary limits. The API doesn't specify this, because those are "management" limits so to speak, that can be changed by admins and so on. It's like cgroups limiting your RAM usage despite the max ram being reported higher in a way.

    It should gracefully handle it though but it's hard.
    Yes, I know, however that is what the discussion was originally about (when you quoted someone else, but accidentally made it look like I wrote that), at least as I understood it. The wine-server can't deadlock itself with internal locks insofar as it isn't using any itself. It just manages locks on behalf of the Windows-apps. So that might cause the Windows-app threads to deadlock, or more likely, they might lock themselves, but not the wine-server thread.

    However ntsync code can theoretically deadlock itself with internal locks (one of them being that "wait-all-lock" which was discussed) because they guard memory that is used by multiple threads inside the Linux kernel ('shared' memory that isn't declared as such because it is used inside the kernel only). When two Windows apps call ntsync functions at the same time, there are two threads inside the Linux kernel using ntsync code. So now the ntsync code could deadlock itself theoretically, using internal locks, if that code uses these internal locks incorrectly.

    Probably that sounds more complicated than it is....

    EDIT: The reason the same functionality can't deadlock by internal locks, is that the Wine-server's interface serializes the functionalty into a single thread, so it doesn't need locks anymore (at the expense of not being able to use more than one core at the same time, for this functionality).
    Last edited by indepe; 29 January 2024, 05:06 PM.

    Leave a comment:


  • Weasel
    replied
    Originally posted by indepe View Post
    Not a internal-lock-too-big problem. A completely single-threaded server wouldn't even need internal locks (insofar as really all operations are carried out on that thread only).

    That would come into play only if and insofar as some operations are carried out by clients or async callbacks or so (which might be an optimization in an event server of other design than the wineserver, unless those are using so-called lock-free programming.)

    ntsync needs locks because those operations within the kernel are multi-threaded (and not using lock-free programming, which would most likely be very difficult in that design)
    It's not about internal locks, it just doesn't account for arbitrary limits. The API doesn't specify this, because those are "management" limits so to speak, that can be changed by admins and so on. It's like cgroups limiting your RAM usage despite the max ram being reported higher in a way.

    It should gracefully handle it though but it's hard.

    Leave a comment:


  • oiaohm
    replied
    Originally posted by Weasel View Post
    For (3), it's a bug because they didn't expect the kernel to reach the limit and block itself. It was not accounted for, not intended.

    It's a mistake that is hard to fix, but still, a mistake. A bug. It's not a design flaw, it's the kernel's "quirk" (imposing limits not in the API itself).
    Yes as said from the start agreeing with you is the deadlocks wineserver suffers from are bugs normally in something else.

    Not accounted for is correct. Yes not intended is also could be correct. There is a little miss. These issues with kernels having some quirk causing wineserver to deadlock get reported when reduced to unique cases about 10 times a year. Out of these cases less than 1 percent is a full kernel stop. This means that for 99%+ of these issues a watchdog process would be able to pick up that wineserver has stopped and step in prevent everything from going out of control. Kernel full stop the system already dead.

    This is a mistake you can in fact design to mostly counter by adding a watchdog.

    Car crashes are hard to account for yet we still build in seat belts and airbags.

    Kernel blocking itself and scheduler then not giving thread any more CPU time happens more than we like and it lot more common of a issue.

    I'm working in it company where I have to do one work manually everyday that I'm willing to automate We have to monitor taskbar in every two hours for a process, if process have zero CPU utilizat...


    Please note OS deadlocked process(type 3 problem) is not a Linux/BSD/Unix/Macos unique thing when you look into windows stuck processes you find that this happens on Windows as well as a rare event..

    Thing is a wineserver deadlocked by any cause can equal the windows PE applications going into totally stupid code paths. Why this comes a major design issue is when Windows PE applications wine is running go out of control because the wineserver is deadlocked.

    Minor design change adding a watchdog to detect when wineserver has stalled and to terminate all applications connected to that wineserver is the counter to 99+% of the reported cases of type 3 deadlocking.

    Weasel any deadlock mitigation that does not have watchdog of some form is not designed to control deadlocks to the max possible. MS Windows does have a few watchdog against type 2 and 3 deadlocks but they are not perfect.

    Wine aiming to be bug to bug compatible with windows it really should have some anti- type 2 and type 3 deadlock watchdog.

    Weasel like it or not this is a design flaw. Yes a design flaw because you have failed to account for something in the design.

    Getting deadlocked without way out due to OS quirk majority of cases this is a design flaw you forgot in your design to allow that OS kernel can have quirks that result in processes not keeping on getting CPU time. For servers stalled out this way without watchdog is particular bad.

    Leave a comment:


  • indepe
    replied
    Originally posted by Weasel View Post
    For (3), it's a bug because they didn't expect the kernel to reach the limit and block itself. It was not accounted for, not intended.

    It's a mistake that is hard to fix, but still, a mistake. A bug. It's not a design flaw, it's the kernel's "quirk" (imposing limits not in the API itself).
    Not a internal-lock-too-big problem. A completely single-threaded server wouldn't even need internal locks (insofar as really all operations are carried out on that thread only).

    That would come into play only if and insofar as some operations are carried out by clients or async callbacks or so (which might be an optimization in an event server of other design than the wineserver, unless those are using so-called lock-free programming.)

    ntsync needs locks because those operations within the kernel are multi-threaded (and not using lock-free programming, which would most likely be very difficult in that design).
    Last edited by indepe; 28 January 2024, 06:19 PM.

    Leave a comment:


  • Weasel
    replied
    Originally posted by oiaohm View Post
    This is your problem "by design it should not"

    Weasel there are more than one form of common deadlock.

    1) Multi thread deadlock this is application internal locking being wrong causing a dead lock. Wineserver design is counter to this.
    2) Multi process deadlock. This can be hidden locks on resources. Wine has had cases of people optimizing wine prefixes on network storage running into a dead lock accessing registry because of file system hidden locking so yes it possible for 2 running wineservers to have event that they dead lock with each other.if user has done something odd to the wineprefixes there are cases that wineserver runs into some other application holding a lock on something it need to use also causing this deadlock. So Wineserver design does not prevent this one fully. Some parts of wineserver design limits how this can happen.
    3) Process to kernel dead lock. This is where the syscall in kernel that the application has called managed to end up dead locked so the wineserver thread/process stops because the syscall to kernel never returns. This rare to happen with Linux kernels more common with the BSD/MacOS kernels
    For (3), it's a bug because they didn't expect the kernel to reach the limit and block itself. It was not accounted for, not intended.

    It's a mistake that is hard to fix, but still, a mistake. A bug. It's not a design flaw, it's the kernel's "quirk" (imposing limits not in the API itself).

    Leave a comment:


  • oiaohm
    replied
    Originally posted by Weasel View Post
    That's a bug. I never said it's bug free. But by design it shouldn't. Bugs are obviously not intended.
    This is your problem "by design it should not"

    Weasel there are more than one form of common deadlock.

    1) Multi thread deadlock this is application internal locking being wrong causing a dead lock. Wineserver design is counter to this.
    2) Multi process deadlock. This can be hidden locks on resources. Wine has had cases of people optimizing wine prefixes on network storage running into a dead lock accessing registry because of file system hidden locking so yes it possible for 2 running wineservers to have event that they dead lock with each other.if user has done something odd to the wineprefixes there are cases that wineserver runs into some other application holding a lock on something it need to use also causing this deadlock. So Wineserver design does not prevent this one fully. Some parts of wineserver design limits how this can happen.
    3) Process to kernel dead lock. This is where the syscall in kernel that the application has called managed to end up dead locked so the wineserver thread/process stops because the syscall to kernel never returns. This rare to happen with Linux kernels more common with the BSD/MacOS kernels

    Wineserver does have documented deadlocks of type 2/3 and type 2/ 3 the current design of Wineserver does not prevent or contain.

    Wine support recommendation of one application per wineprefix one of the reasons for this is mitigation against type 2 and 3 deadlocks so that the number of effected applications is kept down.

    Weasel simple thing to forgot is OS today are multi threaded so even if you program is not multi thread it can still be deadlocked because the multi thread OS can deadlock on operations it performing for your application.

    There is a serous reason why watchdog items are good even for single threaded designed applications.

    The reality here is wineserver code can be 100 percent bug free and due it having a watchdog and something else be wrong it end up deadlocked.

    The by design argument is commonly a way not to have a broad enough view of the problem space. You straight up said that wineserver does not deadlock the reality is wineserver does deadlock and they are rarer forms of deadlock.

    Yes there is a myth we make our program single thread we have made it immune to deadlocks when in reality this is only prevent 1 form of deadlock. There are more than the 3 I list for how to end up deadlocked and over wine history the wine project has seen them all happen to wineserver.

    Wineserver current design does deadlock rarely due to the more exotic forms of deadlocks mixed with a percentage of problem exists between keyboard and chair attempting things that should not be done by documentation.. One of the big issues is some of wineserver deadlocks end up trigger applications go complete stupid and start attempting to delete millions of files, consume all ram and so on.

    Wineserver deadlocks are rare so most people are not aware of how much of a disaster wineserver deadlocks can be..

    Weasel lot of developers add try catch statements to programs so that unintended event does not cause absolute disaster. A watchdog on the wineserver would the same thing unintended event happens result in wineserver stalled out the watchdog steps in kills everything running under wine before everything goes totally out of control.

    Good design should be able to handle a decent percentage of unintended events. Particularly the ones that can be highly harmful to the end user like windows applications going completely nuts because wineserver has stopped processing..

    Leave a comment:


  • indepe
    replied
    Originally posted by Weasel View Post
    Yeah I'm just unsure what happens when it's contended and not waiting because it's busy doing something else for the moment (single thread issues you know). Does it switch that core later when it goes to wait?
    If/while it is doing something else, a work item would get added to a lock-free queue. If you want to protect against rogue applications (which in the case of wine you would), it would be a separate queue for each such app, for example. However, these are implementation-dependent details, which would depend also on the experience made during prototyping/testing.

    Leave a comment:


  • Weasel
    replied
    Originally posted by oiaohm View Post
    This is when someone goes and incorrectly presumes how things go wrong.

    Does wine server always proceed to the next task when queued when blocked. The answer is a horrible no it does not.



    There are a stack of different ways Wineserver single thread can stop dead.
    That's a bug. I never said it's bug free. But by design it shouldn't. Bugs are obviously not intended.

    Leave a comment:

Working...
X