Kernel Patch Revved For Syscall User Redirection To Help Newer Windows Games On Wine

[oiaohm]

I would like to see Wine running Windows x86-32 code in Linux x86-64 mode - maybe the extra CPU registers available in 64-bit mode can be used to speedup the x86-32 code by 2%.

The wine developers are already doing this as a side project. https://github.com/AndreRH/hangover x86-32 code running on x86-64 mode works out about 5 percent slower if it works at all. Lot of the programs using direct windows syscalls are unlikely to ever play well with hangover will be explained once I have pointed out what you got wrong in the following.

Well, but binary translation is quite trivial to implement in the beginning: The translator parses assembly instructions in 4K pages which triggered a page fault and replaces original jump/call targets with its own jump/call targets. Just code needs to be translated, data is not being translated at all unless the application reads/writes from/to pages containing the translated code (such reads/write are unlikely because the translated code pages are being allocated at addresses making such conflicts improbable).

Except there is one problem. The programs using windows syscalls will also contain the real horrible. This is where the assembly instructions are on read, write and execute pages and self modifying code will be expected.

https://shanetully.com/2013/12/writing-a-self-mutating-x86_64-c-program/

Yes this horrible stuff where code is modify calls and jumps on the fly. With extra bonus that the modification can be a value received from remote server so you cannot pre solve this. So the idea of translater parses assemble instructions in 4K pages basically does not work. Also processing in single instructions will cause code to time out.

Binary translation is not trivial to implement on the programs under windows using syscalls directly these are true evil applications. The application directly using windows syscalls don't care how much the weaken system security by using rwx(read write and execute) pages. They at times intentionally do a code modification to cause cpu to have to flush L1 because the cpus own branch predictor has gone down the wrong path and expect to see this stall this also means if you solve this horrible out application can still break.

The common reasons for doing the horrible of direct windows syscalls is anti-cheat. Common reason for self modify code in binaries is anti-debugging that is used to protect anti-cheat systems in clients in the developers mind from being simple to bypass. Once you have self modifying anti-debugging code to deal with binary translation gets a lot harder. Please note the anti-debugging code can be so touchy that true Windows users have the applications not run and have to be refunding for games they bought. Then you add in the different performance pattern of binary translated code welcome even less success at running.

The horrible reality here is the ideal applications with wine to be using dynamic translation with will not be the ones using direct Windows Syscalls. Your ideas are fine for the windows applications don't directly use windows syscalls and are not stacked with horrible self modifying anti-debuging code yes this turns out to be over 80% of the windows applications in existence.

Please note is over 99% of the applications directly using windows syscalls have anti-debugging code that is the self modifying type. So the bar to binary translate this class is not your basic or simple to perform binary translation but your insanely hard and complex binary translation to pull off with tones of pit falls. Yes some of these it will be simpler to write a clone game engine than work around the anti-debugging stuff in the existing engine.

[Weasel]

Well, but binary translation is quite trivial to implement in the beginning: The translator parses assembly instructions in 4K pages which triggered a page fault and replaces original jump/call targets with its own jump/call targets. Just code needs to be translated, data is not being translated at all unless the application reads/writes from/to pages containing the translated code (such reads/write are unlikely because the translated code pages are being allocated at addresses making such conflicts improbable).

You realize code is data, right? And yes ofc the application reads its own code ("data") to make sure it hasn't been tampered with, considering the entire syscall bullshit is due to anti-cheat in the first place.

[Weasel]

Except there is one problem. The programs using windows syscalls will also contain the real horrible. This is where the assembly instructions are on read, write and execute pages and self modifying code will be expected.

It's not even necessary for them to be write or self-modifying code at all. While that's indeed the case for some anti-cheat, it's very common for them to check that the in-memory image is the same as on disk (that's the entire reason for moving to PE files in the first place, esepcially ntdll).

Guess what happens if you modify the in-memory code image and replace the syscalls? Mismatch with what's on-disk. And then the anti-cheat crashes the app.

[oiaohm]

It's not even necessary for them to be write or self-modifying code at all. While that's indeed the case for some anti-cheat, it's very common for them to check that the in-memory image is the same as on disk (that's the entire reason for moving to PE files in the first place, esepcially ntdll).

Guess what happens if you modify the in-memory code image and replace the syscalls? Mismatch with what's on-disk. And then the anti-cheat crashes the app.

To perform memory validation you don't need to use direct syscalls. Punkbuster for example does not yes it using a custom kenrel module. There is no need really to go outside Microsoft development guidelines to make a solid hard to beat anti-cheat system. Directly using syscalls is outside Microsoft development guidelines. Once they have broken guidelines they normally break do not use self modify code as well

The reason for the files need to be PE is not checking that the in-memory image is the same as on disk because if they did that one of the historic work around of just putting the PE/ELF in the dll location. The reason why wine need PE at least to some amount is the horrible reality that some anti-cheat systems have their own PE loader and of course if it sees ELF format is dies. This is not checking that memory and disc is the same this is that they don't use the windows provided library loader or are modify the provided dlls in memory.

Basically this stuff gets really horrible because there are a stack of things that you can do for security that no good. Some of those applications with custom PE loaders are load the complete libraries int rwx memory segments so completely losing the NX bit protection on what they load.

So the reason for moving to PE is not because applications are comparing between memory and disc. The applications comparing that memory to disc that the PE stubs of wine worked around that as well so Weasel you need to be asking why were the PE stubs not good enough the answer is not good. The ones that need a real full PE ntdll in wine are all in the horrible class of either own PE loader that is normally not secure or own PE modifier. Yes PE modifier load a ntdll into memory and use Microsoft injection code to patch it in memory so X function does something different to Microsoft documentation we are not talking just checksum the memory this kind of makes debugging/disassemble harder.

In fact just checksum the memory wine providing PE format dlls does not fix. Take punkbuster for example it uploads the checksum values to their servers and since they don't recognise custom built wine having own PE format dlls don't help at all. There is a complete different issues to check that in memory image is the same as on disk and the problem child applications this is no the problem. Some form of self loading or self modification is the problem. Some of the self loading get really questionable because it looks like at times their server is providing the ntdll they are using as in get windows version from client sent it to server and have server send back own version of ntdll with own tweaks.

Just a note: Unless the application is communicating with a remote anti-cheat digital-rights-management server, the binary translator can make the translated code see the timestamps it wants it to see. If it is communicating with a remote DRM server, then the server has to account for the fact that network speed is an imprecise interval, is not a single precise number, and the binary translator can exploit this fact.

Issue here is for multi player play that is some what fair the game servers just like NTP time servers clients have methods of dealing with that the network is not a precise interval so its a solve-able problem.

Once you start having to fake time information as well things get a lot more complex you are no longer talking a basic generic binary translation but a unique per application binary translation because areas you will need to modify to make the anti-cheat/DRM happy in time there will be others with game play and application activity that you cannot mess with time without screwing up how the application performs for the end user.

The applications using direct syscalls are also interfacing with a remote DRM or anti-cheat server over 99% of the time. Basically as soon as see applications using direct windows syscalls all the horrible are fairly much there. Some implemented better than others.

The really warped is most of the games using anti-cheat systems that use code that does direct syscalls as soon as wine developer or anti-cheat cracker makes the game work the game developer releases a patch to all users removing that anti-cheat system due to the random failures the anti-cheat system is causing on non cheating users. This also makes these miss behaving pains in the but mostly be games that have been out for less than 12 months. Now wine having better support out the box for these pain in but using direct windows syscalls will basically see less applications released using direct windows syscalls. So the could possible be a feature wine implements that in 5 years time we have absolutely no use of and only need to be there so we don't need it.

[oiaohm]

Thanks for the lengthy post.

Note 1: In 5 years, DRM methods will be more advanced/complex than today, not less advanced, so I think it is unlikely that today's Wine new features will be of no use 5 years in the future.

With the direct syscall the other horrible thing those using direct syscalls is doing there is a reason why its going the way of the dodo.

Anti-virus software need really to be able to scan applications right to work out of they are hostile or not. Yes these horrible anti-cheat/drm systems give increase risk of false positive by anti-virus resulting in having to give refund/possible bad press.

As the server side protections to DRM and anti-cheat get better the client side solutions to DRM and anti-cheat come more of way to lose customers so not profitble. So its quite likely in 5 years time that the direct syscall feature is basically not use by wine because the modern more advanced/complex solutions don't require it and give a larger customer base.

The worst part about all these anti-debugging moves is those developing cracks to anti-cheat and DRM have fairly much worked out how to defeat all your client side stuff. Companies making the client side stuff for anti-cheat and DRM have been stacking more and more of the horrible old methods together to attempt to fix that almost one of the client side stuff works against determined cheaters resulting in making non cheating players life harder and company buying their product having to perform more refunds.

Something to be very aware of as well if you attempt to put out a mobile game with these horrible nasty anti-cheat solutions you submit it to google or apple to go into their applications stores guess what your program gets rejected. So this need server side anti-cheat. Now what about your next generations Playstation and Xbox basically the same as you phones that it has to be server side anti-cheat and well behaved anti-cheat.

PC game maker is just a few years behind.

Note 2: The performance of CPUs in desktop/notebooks has higher variability than performance of CPUs in consoles, so it is questionable whether a remote DRM server communicating with a desktop/notebook PC would be able to recognize that the Windows application is running in a binary translator with a 2x slowdown.

This is your being clueless on how sensitive the detection is. Some game companies servers ban you for running in virtual machine using a full server side anti-cheat system. So less than 5 percent different is overall performance is enough to get you banned as it puts you outside the profile of a know bare metal run and into a known running on virtual machine.

So not only do you have to do binary translation the execution has to work out to be as close as possible to running on bare metal. Binary translation comes a lot more complex problem. This is why so much effort to avoid binary translation.

[oiaohm]

I don't understand why you believe that I am "clueless" because you have not addressed in your posts basic ideas so far, such as: The binary translator can make the translated binary believe that it is running on a slower CPU with a CPUID fabricated by the translator. The translator cannot create a faster CPU than the CPU on which it is running, but fabricating a slower CPU is certainly possible. The translator can also fabricate a network with a smaller bandwidth and/or with a higher average latency and/or with a larger latency variance/spread.

This is all clueless. The reality is it insanely hard to exactly emulate a modern x86 CPU. Yes a binary translator might be able to lie about the CPUID but will it make the right "CPU execution time jitter". Yes back in 2015 https://lwn.net/Articles/642166/ people mucked around with this as a random number source. Guess what when you know exactly what should be executing the model of CPU you can make a predictable jitter model. Yes this is allowing that X execution of X instructions should produces high heat at this point so should cause a slow down no matter how good your cooler is and so on its not 100 percent there is a percentage of random in there but there are clear patterns that particular models of CPU should do. Fun for binary translation is putting different x86 instructions though the cpu so will generate a different jitter pattern. Yes due to x86 cpus having jitter is not really possible to emulate a different x86 cpu jitter because the host x86 CPU jitter will screw you over. To emulate a CPU jitter you need a CPU that can do perfect hard real-time but then you are also talking about also need a clock speed at least 40 times faster than the cpu you are emulating as well. Remember the games this modern technology to detect cheats have a min cpu requirement at least of 1Ghz so where in hell are you going to get a 40Ghz processor in real-time class. This is why you are clueless what you are suggesting really cannot be done.

The translator can also fabricate a network with a smaller bandwidth and/or with a higher average latency and/or with a larger latency variance/spread.

This idea is stupid. One of the common cheats against games has been people messing with the speed of their networks. So doing what you described is a path to getting self banned.

https://kb.meinbergglobal.com/kb/time_sync/time_synchronization_accuracy_with_ntp

Yes games are normally using NTP server/client in kind of reverse. Yes there is a path to average measure network and cpu that is send out packets over time and be tracking the response time. Artificially slowing network down in emulation will normally generate something way to clean by again removing natural jitter. Remember a lot of internet connections are under 1Mbps so the network side anti-cheat is looking for performance artefacts that your cpu makes that will be visible on 1Mbps or slower network connection. By the time you get your connection slow enough that the network side anti-cheat could not see anything to ID CPU and Platform the game server will be disconnecting you from game because you will be timing out.

Horrible reality here is a network based anti cheat to id your cpu make and model does not need CPUID value or to run any instruction itself locally it only need to monitor the jitter patterns. So there is a lot more than just emulate the instructions or rewrite the instructions to get past anti-cheat systems.

[oiaohm]

This discussion is infinite because every time I or you send a message then the other person will send a response which raises the complexity in order to beat all the previously sent messages.

Not at all.

https://www.pcgamesn.com/overwatch/overwatch-linux-bans

atomsymbol how do you think the bans here where generated. Players are getting told they were running a chat program and that is why they were banned when it really was dxvk and wine. This is because companies like Blizzard are not in fact looking at the software running on machine they are looking at the jitter from there network server side anti-cheat. If you are inside a particular operational jitter pattern you are not cheating if you are outside that jitter pattern you are banned for cheating and depending on the patter of jitter you show is what the message tells you that you were doing to get banned. Yes this is NTP server to client stuff in reverse.

The reality here atomsymbol like it or not you ar really clueless about the network anti-cheat technology and how sensitive the stuff is. Yes the false positive events and the fact that the network anti-cheat can sort out virtual machines from baremetal and also will ban people using qemu usermode to run wine.

atomsymbol you have been writing a stack of possible theory problem here I have been doing support and I have see what companies in fact ban and at times their developers do give basic over views of their anti-cheat on their servers to help wine developers out. The ideas you are suggesting like dynamic binary translation that what qemu usermode around wine does and that trips the anti-cheat systems. There is no simple fix because its this jitter technology you are up against. Yes is not 100 percent perfect yes it bans players ever now and again who have done nothing wrong and lets a few though that are up to no good but over all it fairly effective.

Its really easy to suggest a idea that basically does not work in real world because there are a stack of real world factors that the idea has not allowed for. The idea that you can just binary translate your way around these problems is that. Hangover developers with wine have also got themselves ban from particular games as they have been trying to get hangover to work. Some games the network anti-cheat will make it impossible to run them in hangover so those games will not work on arm based devices including future apple computers. Binary translation unless developers of some games decide to change their network anti-cheat its not going to be an option or at least a option you cannot use as a generic fix all.

I have more of a question how many times will you attempt to argue the same solution until you wake up you are looking at the problem way to simple without the require information on how the protections work. The anti cheat protections the network side ones looking for particular runtime jitter stuff is a very hard barrier. This means you want the least amount of emulation. There is a long long list of people who have attempted the ideas you are suggesting only to end up banned from particular games and upset.

[oiaohm]

- I don't play the particular kind of games you mentioned. If it is larger issue for players who play those games then they should team up and initiate a class action lawsuit.

People don't read the terms of service. Using a not offically supported platform to play game is also technically by the way they word license breach of software license. So the software vendor has is ass covered. Class action is not possible.

As you mentioned, it is probable that a Wine developer would get his/her account disabled while trying to implement a binary translator that works for an online DRM Windows game of Linux. Getting banned after the development phase ends is less likely. The entry cost is much larger than the cost of use after entry.

The idea of getting banned after development phase end is not less likely when the game developer is no changing the conditions for banning.

Have the Wine developers tried to compile Wine in Windows and use native Windows as fallback for cases they cannot yet handle in Wine and use that as a starting point so that they first can gather statistical data using close-to-zero-overhead methods about what the game actually is doing and only after they know that is going on they start working on supporting the game in Wine Linux?

What you are writing here has been done for over 20 years with wine. There is a reason why a large section of wine can be built for windows and run on windows yes this is for compare testing.

This approach seems much more logical than going the route of using just Linux or qemu on games that you know will get your account disabled.

Qemu route at different points in history there have been no other choice. Think powerpc apple computers of the past there is no x86 processor there. Or arm processors and people wanting to run windows applications. Again you have no choice but to use on fly binary translator to support as many applications as possible and qemu provides a decent one.

Wine developers started really cursing when OS X removed 32 bit x86 support because they knew this would bring having to perform binary translation and the issue this is going to bring.

Then the developers who chose the wrong route are wondering why their accounts got disabled.

No the wine developers due to having development accounts with a lot companies know exactly what is going wrong. Also they can get there account undisabled without any argument because with a lot of the providers they can login and just self press the unban/undisable button. So minor trouble to the wine developer to person who buys game lot more trouble.

As I already wrote, this discusssion is infinite. I wonder how are you going to overthrow my argument about using Wine from Windows as a starting point. Yes, it is all "just a theory" ... but maybe having a solid theory actually matters in real life ...

No really that point was you being clueless you were just stating what the Wine developers already do. Reason why on the bug report with implement syscall support over head of emulating the windows syscalls is a factor. Windows syscalls are not as effectively done as Linux ones so there is some wiggle room not a unlimited ammount.

Your post seems to be suggesting that those developers were using their main email accounts instead of separate email accounts created specifically for such a purpose.

Of course the wine developers are using their main wine development accounts for game testing. Something that you are not aware of is that active developers get special free accounts where they can install all software from a particular providers without paying a cent. Specials accounts that are 1 provided for free by the game developers 2 are know to be trying experimental things 3 that have the anti cheat systems lighted that not to go off when being used with windows in VM. Yes there are many games out there where installing Windows in a VM and the installing game and attempting to play it results in banned unless you hold a special account of a Wine developer or a internal company developer.

Really a wine developer account getting banned by anti-cheat does not bode well at all for end user.
1) the anti cheat on the wine developer is lighter than a person who bought the game.
2) wine developers don't normally run games for that long. Longer game runs more of a jitter profile can be built.

Basically once a wine developer account with these companies is getting banned doing something end users are totally screwed attempt to run that game. If game works with wine developer account end user still could be screwed.

[Weasel]

Well oiaohm's imagination is as active as ever, some things never change I guess. (he's right that binary translation wouldn't work, but the reasons he states are mostly made up, either by him or other people with not much clue)

[oiaohm]

20 years is one generation. 20 years is usually a time when old ideas get reimplemented again because a new generation of developers takes over a project or creates a project without knowing that a similar project already existed 20 years ago.

Except this is wine. Wine still has its same project lead from 20 years ago. Wine project has not done a generational change in core developers yet. The current wine project leaders are most likely going to be there for another 20 years. There are quite a few open source projects that run on 40 year cycles as a generation turn over.

I am sure that many of ideas that you believe are "new" and "untried" have already been thought of in 1950-1960 when computers where the size of whole rooms.

When you have a project like wine with a very slow turn over of core developers its really hard to be thinking up some new idea they have not tried at some point and if the idea is decent it remains in usage. Lot of cases that idea will be in active usage. Wine testbot running test suite of wine against windows to compare the differences that started 20+ years ago and is fairly much being done the same way as 20 years ago with a few upgrades.

What issues do you mean exactly? The DRM server to which the old x86-32 application used to connect is no longer operational.

I have said anti-cheat a lot and you keep on returning to DRM. Network side anti-cheat you find on old dos games running in dosbox and if you don't use the game makers provided dosbox you get banned as well. Turns out game sales drop when you have excess number of cheating players in a particular game even of old games. DRM may be removed from old games but the anti-cheat stuff does not go away.

Well oiaohm's imagination is as active as ever, some things never change I guess. (he's right that binary translation wouldn't work, but the reasons he states are mostly made up, either by him or other people with not much clue)

Funny enough no winehq hangover project is using binary translations and is hitting particular issues. The issues are a little different to the theory issue from those who have in fact tried todo it for real Weasel. The network anti-cheats doing jitter measurements that was not really known until hangover started doing what it was doing. There were a few oddities where people would get a dos game using dosbox swap in their own dosbox and get banned before the hangover work. Yes the jitter method explained what was going on with users changing dosbox on particular games as well.

This is the second time wine project has been doing binary translation powerpc for mac to run x86 windows binaries was the first back then the jitter stuff did not exist. Weasel have you looked into this in the last 4 years. That is important a new harder to defeat barrier has basically appeared.

Weasel the hangover project has been used to run old quake based games. Some of those the developers got banned using some of these old quake based don't have DRM or local anti-cheat at all because they tripped the network anti-cheat. So your claim is that binary translation would not work is not exactly true. Particular games and programs binary translation with wine will work and does in fact work. Most of the games using anti-cheat using syscalls are also using network jitter anti-cheat as well that method kills binary translation unless you can get the developer of game to add a new acceptable profiles for the translation. Yes this is a new bar need to get over that did not exist when wine was using qemu on ppc on mac to run programs.