The 2010s Were Very Successful For Wine Thanks To CodeWeavers + Valve's Steam Play

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts
  • duby229
    Senior Member
    • Nov 2007
    • 7779

    #21
    Originally posted by oiaohm View Post

    Wine test suite is not synthetic. The wine test suite is black box. As it the test in there are based off recorded behaviours of real windows applications. Its lot faster to run tests than full applications.


    Basically this statement is completely false because you don't understand how wine regressions happen 99.9 percent of the time.

    No the reality is wine sucks that anything runs is a miracle. Lets look at some numbers instead of your crap guess work.
    https://test.winehq.org/data/497b9ed...dex_Linux.html

    There is 675 unit tests. 19 error as in crash as application. 361 report not functioning correctly maybe good enough. 2.8% don't work at all. 53.5% might work for some applications. The result is only 43.7% of wine by its own test suite in fact works properly.

    Since wine test suite is based on real applications there is a lot of broken there. Wine is in it stable release process for the year. This is when a huge stack of applications are run against wine to detect if the test suite is right wrong or indifferent as well. You will notice that winetricks has a huge stack of applications it can in fact install this is for the yearly lets install a stack of real applications and check.

    The horrible reality here most(99.9% of them) of Wine applications regression is that something in the 56.3% in fact gets implemented correctly now something else in the 56.3% that is not implemented correctly that has not been problem now is because since the part has been implemented correctly the application code path changes and it now wants to use more functions including the stuff that don't work. This is the true nightmare game of pick up sticks.

    Wine is not even half complete that a surprise to most people for how many programs in fact work.
    You already know I disagree with you. I'm not going to humor you again. The bottom line fact is that Wine's unit tests -ARE- synthetic and as such they -CAN'T- test behavior of current -actual- apps.

    EDIT: No, the biggest flaw with Wine is that Windows interfaces exhibit multiple different behaviors depending entirely on what and how it is being used and Wine makes no attempt to emulate that fact. They don't test Wine with -any- actual apps to establish what behaviors -do- get exhibited and instead they implement one behavior that may work in one scenario but not another and they have no possible way to know any better because of that.
    Last edited by duby229; 06 January 2020, 12:11 AM.

    Comment

    • oiaohm
      Senior Member
      • Mar 2017
      • 8391

      #22
      Originally posted by duby229 View Post
      You already know I disagree with you. I'm not going to humor you again. The bottom line fact is that Wine's unit tests -ARE- synthetic and as such they -CAN'T- test behavior of current -actual- apps.
      That is false. Wine test suite only tests the behaviours of the applications it was model off. If you are using a application was used to built the test suite model behaviour there is no difference in result between the wine test suite and the real application other than run-time ie the test suite way faster.

      Actual application showing a difference would be ones that the wine test suite was not modelled off.

      Originally posted by duby229 View Post
      No, the biggest flaw with Wine is that Windows interfaces exhibit multiple different behaviors depending entirely on what and how it is being used and Wine makes no attempt to emulate that fact.
      If you go into the unit tests you will see they don't just test a function or part a single way. So yes its testing for multiple different behaviours that have been collected from test runs of multi different applications.

      Originally posted by duby229 View Post
      They don't test Wine with -any- actual apps to establish what behaviors -do- get exhibited
      This is false.

      This time of year with the stable release we see the biggest expand to the wine test suite as part of the stable process every year with wine is grab a pool of real applications and run them against wine. While at it expand the test suite for next year with the behaviours detected from those applications.

      So you claim that wine don't test with any actual apps to establish what behaviour is absolutely false. Ok you might have a complaint that is only major-ally done once per year to improve the models in the testsuite what is the fact. Doing true application testing all the time with a 2 week release cycle is not possible this is why the behaviours are modelled into the test suite. Testing with actual applications is way slower this is why the stable release process each year is also slower.

      If wine was testing with actual applications all the time max of 6 release a year and a lot slower development.

      Originally posted by duby229 View Post
      instead they implement one behavior that may work in one scenario but not another and they have no possible way to know any better because of that.
      There are many patched stuck in staging because they do exactly described here as in implement only one behavour. When you run that patch with the wine testsuite you see hello its going to break like 30 other applications because its not doing all the behaviours it should.

      The more models added to the test-suite the broader it coverage comes. It would be good if testsuite model work was done more though the year than once a year major push but that is a project resources limitation.

      Comment

      • duby229
        Senior Member
        • Nov 2007
        • 7779

        #23
        Originally posted by oiaohm View Post
        That is false. Wine test suite only tests the behaviours of the applications it was model off. If you are using a application was used to built the test suite model behaviour there is no difference in result between the wine test suite and the real application other than run-time ie the test suite way faster.

        Actual application showing a difference would be ones that the wine test suite was not modelled off.



        If you go into the unit tests you will see they don't just test a function or part a single way. So yes its testing for multiple different behaviours that have been collected from test runs of multi different applications.


        This is false.

        This time of year with the stable release we see the biggest expand to the wine test suite as part of the stable process every year with wine is grab a pool of real applications and run them against wine. While at it expand the test suite for next year with the behaviours detected from those applications.

        So you claim that wine don't test with any actual apps to establish what behaviour is absolutely false. Ok you might have a complaint that is only major-ally done once per year to improve the models in the testsuite what is the fact. Doing true application testing all the time with a 2 week release cycle is not possible this is why the behaviours are modelled into the test suite. Testing with actual applications is way slower this is why the stable release process each year is also slower.

        If wine was testing with actual applications all the time max of 6 release a year and a lot slower development.


        There are many patched stuck in staging because they do exactly described here as in implement only one behavour. When you run that patch with the wine testsuite you see hello its going to break like 30 other applications because its not doing all the behaviours it should.

        The more models added to the test-suite the broader it coverage comes. It would be good if testsuite model work was done more though the year than once a year major push but that is a project resources limitation.
        You can go ahead and add as many thousands of synthetic tests as you want, but as long as Wine refuses to emulate Windows behavior and test against real actual apps it won't make any tiny little bit of difference as actual history has already proven.

        Comment

        • oiaohm
          Senior Member
          • Mar 2017
          • 8391

          #24
          Originally posted by duby229 View Post
          You can go ahead and add as many thousands of synthetic tests as you want, but as long as Wine refuses to emulate Windows behavior and test against real actual apps it won't make any tiny little bit of difference as actual history has already proven.
          The thing you are missing is every with the stable release not the development release is in fact tested against real actual applications. The bug fixes in the stable branch release process are aligning to Windows behaviour.

          The reality here is wine is so far incomplete that implement any Window behaviour is going to break things. It would be different if wine was at like 90% API/ABI implementation instead of roughly 50%,

          duby229 the testing against real world applications is being done right now. About time you go and read the mailing list. Its a once a year thing.

          Comment

          • Weasel
            Senior Member
            • Feb 2017
            • 4497

            #25
            Originally posted by duby229 View Post
            You already know I disagree with you. I'm not going to humor you again. The bottom line fact is that Wine's unit tests -ARE- synthetic and as such they -CAN'T- test behavior of current -actual- apps.

            EDIT: No, the biggest flaw with Wine is that Windows interfaces exhibit multiple different behaviors depending entirely on what and how it is being used and Wine makes no attempt to emulate that fact. They don't test Wine with -any- actual apps to establish what behaviors -do- get exhibited and instead they implement one behavior that may work in one scenario but not another and they have no possible way to know any better because of that.
            Well you are wrong, but oiaohm doesn't know what he's talking about either, so not surprised you get confused.

            Wine tests are basically testing the behavior of Windows APIs. You can write extra tests yourself if you feel like an API has a specific corner case with specific parameters or global state. This is why I said they test the correct functioning of the Windows API.

            Now the issue is that many applications abuse the API in undocumented ways or using low level hacks that Microsoft never documented (btw, the API is documented on MSDN, though not always correct). This is especially true with pre Windows XP applications.

            Of course, since the apps misuse the Windows API, they break on Windows too, not just on Wine. Normally, that is.

            However, Microsoft added compatibility shims for such broken apps which change the behavior of the APIs only if they are ran under the specific broken app. The reason they added it is because, as a business, they want people to upgrade their OS. If an app breaks, who do you think they will blame? The broken app's dev, or Windows?

            Even though it's not Microsoft's fault, they would get the blame.

            Wine tends to not do such shims. They just implement the "correct" API, which is the API without such app-specific shims that change its behavior. You can't properly add unit tests for this anyway, since the Windows API won't recognize you as the broken app.

            Note that you can still tinker with Wine for such corner cases, that's why you have winecfg, dll overrides, and other registry settings. Still, that's up to the user to configure, rather than Wine automatically using a shim here.

            Here's a few examples of the amount of hacks some apps do: https://devblogs.microsoft.com/oldnewthing/?p=41373

            Comment

            • oiaohm
              Senior Member
              • Mar 2017
              • 8391

              #26
              Originally posted by Weasel View Post
              Wine tests are basically testing the behavior of Windows APIs. You can write extra tests yourself if you feel like an API has a specific corner case with specific parameters or global state. This is why I said they test the correct functioning of the Windows API.

              Wine test suite is not just documenting the windows API as such. It also as here documenting the differences between different versions of Windows. So when wine is set to X version of windows it should pass the tests the same in way. Those tests are based off what applications made for those versions of windows expect.

              Originally posted by Weasel View Post
              Of course, since the apps misuse the Windows API, they break on Windows too, not just on Wine. Normally, that is.
              Not quite. In some places(quite a few places) wine is coded to be tolerant to misuse. The broad wine test suite with data from multi versions of windows you see that request to a function can come in a few different forms and then wine version of function accepts them all and the windows version reject something unless shimmed this happens quite a bit. This is going after the one implementation solution. It really rare that this extra tolerance causes any application issues.

              Originally posted by Weasel View Post
              However, Microsoft added compatibility shims for such broken apps which change the behavior of the APIs only if they are ran under the specific broken app. The reason they added it is because, as a business, they want people to upgrade their OS. If an app breaks, who do you think they will blame? The broken app's dev, or Windows?
              Wine is using a different method to shims. You set X windows version that is what you get with wine with some extra tolerance. The reality is shim under windows are attempt to implement what old version of windows did. So you can make the shim process a lot simpler avoiding the complete database problem.

              Think about it all those Microsoft shims are just to implement something a older version of windows did. Testing windows shims might end up coping a Microsoft developer error when they coded the shim so they are pointless to test suite when you can get your hands on real copy of windows to test suite.

              Microsoft changes in behaviour also align to windows version number of some where. Wine path here is simple we use windows version as key on this stuff.


              Originally posted by Weasel View Post
              Note that you can still tinker with Wine for such corner cases, that's why you have winecfg, dll overrides, and other registry settings
              No that is to tinker around where wine is incomplete. Only value in there that should need to be changed for compatibility if wine was complete is the Windows version.

              Reality here Weasel you are just as wrong. Problem here you were thinking shim and overrides was the way forwards. A program to get platinum rating in wine appdb testing the only thing allowed to be changed is the windows version and the application must work perfectly. Anything else is hacking around defects in wine. Platinum rating is aligned with what the wine project objectives are.

              There is more than 1 way to skin a cat. Microsoft has chosen shims and a database of what shims to apply this does come back and bite with increasing load time resulting with windows 10 they had to trim the compatibility database breaking old applications.

              Wine is going after a very much simple method. Applications were always written for particular version/versions of windows so as long as wine gets it right for when wine is set as X version of windows it appears enough like X version of Windows everything is good no shim or compatibility database required.

              Comment

              • duby229
                Senior Member
                • Nov 2007
                • 7779

                #27
                Originally posted by oiaohm View Post


                Wine test suite is not just documenting the windows API as such. It also as here documenting the differences between different versions of Windows. So when wine is set to X version of windows it should pass the tests the same in way. Those tests are based off what applications made for those versions of windows expect.



                Not quite. In some places(quite a few places) wine is coded to be tolerant to misuse. The broad wine test suite with data from multi versions of windows you see that request to a function can come in a few different forms and then wine version of function accepts them all and the windows version reject something unless shimmed this happens quite a bit. This is going after the one implementation solution. It really rare that this extra tolerance causes any application issues.



                Wine is using a different method to shims. You set X windows version that is what you get with wine with some extra tolerance. The reality is shim under windows are attempt to implement what old version of windows did. So you can make the shim process a lot simpler avoiding the complete database problem.

                Think about it all those Microsoft shims are just to implement something a older version of windows did. Testing windows shims might end up coping a Microsoft developer error when they coded the shim so they are pointless to test suite when you can get your hands on real copy of windows to test suite.

                Microsoft changes in behaviour also align to windows version number of some where. Wine path here is simple we use windows version as key on this stuff.



                No that is to tinker around where wine is incomplete. Only value in there that should need to be changed for compatibility if wine was complete is the Windows version.

                Reality here Weasel you are just as wrong. Problem here you were thinking shim and overrides was the way forwards. A program to get platinum rating in wine appdb testing the only thing allowed to be changed is the windows version and the application must work perfectly. Anything else is hacking around defects in wine. Platinum rating is aligned with what the wine project objectives are.

                There is more than 1 way to skin a cat. Microsoft has chosen shims and a database of what shims to apply this does come back and bite with increasing load time resulting with windows 10 they had to trim the compatibility database breaking old applications.

                Wine is going after a very much simple method. Applications were always written for particular version/versions of windows so as long as wine gets it right for when wine is set as X version of windows it appears enough like X version of Windows everything is good no shim or compatibility database required.
                Wow, wall of -bullshit-....

                Comment

                • Cybmax
                  Phoronix Member
                  • Jan 2019
                  • 92

                  #28
                  Interesting read tho, but there is no such thing as "getting it right", cos M$ cant always do that either - ie. bugs. There are multiple scenarios where game's break due to windows updates... why is that? Cos M$ made a bug? Well.. sometimes ofc, but most times it is because they fixed something. So the game dev's need to update, or ppl figure out "hacks" (ie. replace dll's or whatnot).
                  The second part of the above scenario is ofc what RESULT you would have if a game/app does something "wrong". Running on Windows nothing might happen... maybe you loose a few fps.. maybe some minor thing happens. In Wine, a lot of the times, a "minor" bug will cause a crash and a 100% non working app/game. This is what ppl (like me) mostly get a bit annoyed by.

                  Troubleshooting this on a case of "But this works fine with windows X!" will result in the dev's reply of "But this is the wrong behavior! This is a game bug, and not our fault". Probably because some game/app dev does something wrong using whatever API, that will NOT cause a total meltdown in Windows, but WILL with wine.

                  So the testsuites that are being updated nowadays are not "real programs", it is what i like to think of as "expected behavior" vs api calls. Wine dev's do not run the testbot with 1000'nds of programs to see the output... only doing that WOULD be a REAL testcase. M$ do not either, other than their own software i guess (Office and whatnot).

                  The only way past problems like this would be to not make a "one shoe fits all" solution (that current Windows 10 are not able to do either as pointed out above). It is not PURELY because of creating a market for buying the newest and greatest, cos there are huge development costs involved in catering to everyone. Just see the uproar created by MacOS going pure 64-bit, and now Ubuntu flagging the same.

                  It is more or less impossible to move forward catering to everything that has ever existed. Just cos there are 2-3 ppl playing 16-bit "pong" someplace in the universe, does not mean this should work forever and ever. If i was to vote, i would vote "Right, wanna play pong? Then you need to download wine-3.02" instead of how it is now where it is expected to work with wine-5.0. (Yeah, i have no idea if "pong" works.. it is an example. Just so (linux)ppl wont go ballistic claiming i ask something untrue!)

                  It would be an interesting question seen answered i guess: Would dropping "old shit" make it easier to make "new shit" more compatible with wine? (Or what interests ME the most - make for better game performance).

                  Comment

                  • Cybmax
                    Phoronix Member
                    • Jan 2019
                    • 92

                    #29
                    And as a wee tidbit, i ran the winetest64-latest.exe from http://test.winehq.org/data/ on my windows 10 box, and the results was:
                    "58 Tests failed. There is probably something broken with your setup. You need to address this before submitting results."

                    It is nothing special with my windows 10 box. Build 1909. Yet, only 7 less failures for a real windows install than the latest test from the buildbot. Granted, the win10 testfailure lists from the bot runs there vary a great deal depending on whatever patch submitted, and for 5.0-rc4 it was 94.

                    This begs the question: Does 58 tests fail on a real windows install because i actually have something wrong with my setup, or does 58 tests fail due to bugs in windows... or ultimately does 58 tests fail due to the tests themselves are buggy? I dunno if it would be used for a comparison if some report had been delivered, but IF it is meant as a compare "real" vs "wine", it might skew the results if it can't be delivered

                    I am not saying "winetests are stupid", cos it does test various api functions.. it just does not do a "real-world-test".

                    Comment

                    • duby229
                      Senior Member
                      • Nov 2007
                      • 7779

                      #30
                      Originally posted by Cybmax View Post
                      And as a wee tidbit, i ran the winetest64-latest.exe from http://test.winehq.org/data/ on my windows 10 box, and the results was:
                      "58 Tests failed. There is probably something broken with your setup. You need to address this before submitting results."

                      It is nothing special with my windows 10 box. Build 1909. Yet, only 7 less failures for a real windows install than the latest test from the buildbot. Granted, the win10 testfailure lists from the bot runs there vary a great deal depending on whatever patch submitted, and for 5.0-rc4 it was 94.

                      This begs the question: Does 58 tests fail on a real windows install because i actually have something wrong with my setup, or does 58 tests fail due to bugs in windows... or ultimately does 58 tests fail due to the tests themselves are buggy? I dunno if it would be used for a comparison if some report had been delivered, but IF it is meant as a compare "real" vs "wine", it might skew the results if it can't be delivered

                      I am not saying "winetests are stupid", cos it does test various api functions.. it just does not do a "real-world-test".
                      Great post btw...

                      This is what I'm talking about, wine's unit tests do not test to see if wine API behavior emulates MS API behavior. What they are testing for is wine's imaginary "correct" behavior even though there is no damn such thing. The only thing that's real is -actual exhibited- behavior and that varies app by app. If wine is ever to stop regressing horribly every single release then they need to start testing actual real behavior with actual real apps and games.

                      EDIT: And this is the whole reason why appdb rating system is such nonsense. An app might be able to get a gold or platinum rating with one version of wine on one distribution using one graphics driver, but then it gets completely broken on the next wine release or distro upgrade or driver update. Just because an app has such a rating on the appdb doesn't mean it's really gonna work. It's only gonna work if you simulate the exact scenario where it got that rating and no other.
                      Last edited by duby229; 07 January 2020, 05:07 PM.

                      Comment

                      Working...
                      X