The 2010s Were Very Successful For Wine Thanks To CodeWeavers + Valve's Steam Play

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts
  • oiaohm
    Senior Member
    • Mar 2017
    • 8391

    #31
    Originally posted by Cybmax View Post
    And as a wee tidbit, i ran the winetest64-latest.exe from http://test.winehq.org/data/ on my windows 10 box, and the results was:
    "58 Tests failed. There is probably something broken with your setup. You need to address this before submitting results."
    You need to click one more.


    Originally posted by Cybmax View Post
    It is nothing special with my windows 10 box. Build 1909.
    the automated test has not been running that version of windows.

    Originally posted by Cybmax View Post
    This begs the question: Does 58 tests fail on a real windows install because i actually have something wrong with my setup, or does 58 tests fail due to bugs in windows.
    Welcome to hell. A luck install of Windows 10 will run the winetest suite with less than 2 bug. In theory there should be a install out there somewhere with the magic 0 that is currently not in the wine automated testing pool.

    58 failures on a single system is way higher that the wine automated testing so showing. 51 was merged results with the highest per system run only being 30 so 58 is 28 higher than expected. Either Microsoft has broken a lot more with build 1909 that would not be a surprise or your windows driver combination is just bring out a real horrible like a broken installed driver.

    Every one of those 58 test failure at least 10 applications that will not work. At the that is at least 580 applications that will not work on your system. If you use any of that 580+ applications that is another question.

    Odds that windows compatibility shims fix any of the detects ABI bugs by the wine test suite in windows less than 1 in 1000.

    Originally posted by duby229 View Post
    This is what I'm talking about, wine's unit tests do not test to see if wine API behavior emulates MS API behavior. What they are testing for is wine's imaginary "correct" behavior even though there is no damn such thing. The only thing that's real is -actual exhibited- behavior and that varies app by app. If wine is ever to stop regressing horribly every single release then they need to start testing actual real behavior with actual real apps and games.
    No the windows API exhibited behaviour varies install to install not application to application. Wine testsuite is the way it is because there is no point going down the rabbit hole of attempting to make application work when its the Windows ABI that has broken under you feet.

    The stable releases of wine are tested with real apps and games. There are 4-5 stable releases a year this is slower to allow for that testing.

    duby229 the wine test suite has the behaviours of over 6000 applications. Automated installing 6000 applications is going to take you more than 2 weeks let alone running any tests. Stable releases are only tested with 2000 real applications and that is to keep inside the 2 month window.

    You are saying use real world applications. My problem is how in a 2 week release cycle is this possible. Remember that wine test suite is avoiding have to install 6000 applications+. Horrible part here 6000 applications will not be compatible with each other all the time either.

    Yes you are right that wine "correct" behaviour it testsuite is checking for is kind of imaginary because windows ABI is all over the place. But wine test suite is checking for behaviours we know applications expect and if they don't have those behaviours they don't run.

    Comment

    • duby229
      Senior Member
      • Nov 2007
      • 7779

      #32
      Originally posted by oiaohm View Post

      You need to click one more.
      https://test.winehq.org/data/4f0212c...dex_Win10.html


      the automated test has not been running that version of windows.



      Welcome to hell. A luck install of Windows 10 will run the winetest suite with less than 2 bug. In theory there should be a install out there somewhere with the magic 0 that is currently not in the wine automated testing pool.

      58 failures on a single system is way higher that the wine automated testing so showing. 51 was merged results with the highest per system run only being 30 so 58 is 28 higher than expected. Either Microsoft has broken a lot more with build 1909 that would not be a surprise or your windows driver combination is just bring out a real horrible like a broken installed driver.

      Every one of those 58 test failure at least 10 applications that will not work. At the that is at least 580 applications that will not work on your system. If you use any of that 580+ applications that is another question.

      Odds that windows compatibility shims fix any of the detects ABI bugs by the wine test suite in windows less than 1 in 1000.



      No the windows API exhibited behaviour varies install to install not application to application. Wine testsuite is the way it is because there is no point going down the rabbit hole of attempting to make application work when its the Windows ABI that has broken under you feet.

      The stable releases of wine are tested with real apps and games. There are 4-5 stable releases a year this is slower to allow for that testing.

      duby229 the wine test suite has the behaviours of over 6000 applications. Automated installing 6000 applications is going to take you more than 2 weeks let alone running any tests. Stable releases are only tested with 2000 real applications and that is to keep inside the 2 month window.

      You are saying use real world applications. My problem is how in a 2 week release cycle is this possible. Remember that wine test suite is avoiding have to install 6000 applications+. Horrible part here 6000 applications will not be compatible with each other all the time either.

      Yes you are right that wine "correct" behaviour it testsuite is checking for is kind of imaginary because windows ABI is all over the place. But wine test suite is checking for behaviours we know applications expect and if they don't have those behaviours they don't run.
      You do realize you just made a perfect argument for exactly why Wine needs to be an emulator, right? And no, wine unit tests absolutely do -not- test the behaviors of over 6000 apps. They test for over 6000 behaviors that wine devs made up. The -only- way to test the behavior of over 6000 apps is to actually test those actual 6000 apps....

      And also, a 2week release cycle is a huge part of the problem, don't ya think? It certainly isn't enough time to do much credible work. 20/80 being what it is, they almost definitely average less than 3 days each release cycle implementing API features. And like you say if they need to test 6000+ apps it's not much time to do that either, impossible even. A 2week release cycle is pretty dumb.

      EDIT: I realize I'm getting sarcastic, sorry. But I know I'm absolutely right.
      Last edited by duby229; 10 January 2020, 03:08 AM.

      Comment

      • oiaohm
        Senior Member
        • Mar 2017
        • 8391

        #33
        Originally posted by duby229 View Post
        You do realize you just made a perfect argument for exactly why Wine needs to be an emulator, right? And no, wine unit tests absolutely do -not- test the behaviors of over 6000 apps. They test for over 6000 behaviors that wine devs made up.
        The test suite is checking over 600 thousand behaviours. Roughly 70 unique behaviours per application were added to the testsuite. Yes there are a lot of common behaviours.

        Originally posted by duby229 View Post
        The -only- way to test the behavior of over 6000 apps is to actually test those actual 6000 apps....
        This is a arguement you bring up. Microsoft does not test individual applications when they make windows either they use a huge master test-suite like the wine one.

        Originally posted by duby229 View Post
        And also, a 2week release cycle is a huge part of the problem, don't ya think?
        The reason why the wine project has a development branch and stable branch. Running on two different cycles.

        You need to run 6000+ applications to get the same api coverage as the current testsuite does.

        Originally posted by duby229 View Post
        almost definitely average less than 3 days each release cycle implementing API features.
        If you look at the wine project release cycle for development branch out of the 14 in the release cycle 9-10 days is in implementing API features. So you have 5-4 days to QA right nop that is for packaging.

        The automated testsuite between releases can only run 9 to 10 times before the project is out of resources to run it let alone doing individual application testing..


        Please note that 9 to 10 runs for wine testsuite bundles of API changes are tested at the same time.

        So this means you have less than 24 hours to install and test 6000+ applications to keep the same level of coverage as the automated testsuite. Current level of wine project resources this is simply impossible for the development branch.

        The stable release the wine project does slow down a little the final stage of stable release gives 8 weeks to testing. To go from the .0 stable release to the .1 stable release.


        duby229 basically todo the testing you want you are talking a resource increase at least 2000 times what the wine project has now. Wine project has the appdb and we are looking for more people all the time to test applications and report issues to the appdb to increase wine projects testing with real applications and have recorded data on issues.

        Remember wine is not chasing a stationary target Microsoft and applications are always doing updates.

        duby229 you are saying the wine project should do something but you are not providing any method that it can. You are not allowing that wine project is running at resources limit almost continuously.

        Comment

        • Cybmax
          Phoronix Member
          • Jan 2019
          • 92

          #34
          Originally posted by oiaohm View Post
          duby229 basically todo the testing you want you are talking a resource increase at least 2000 times what the wine project has now. Wine project has the appdb and we are looking for more people all the time to test applications and report issues to the appdb to increase wine projects testing with real applications and have recorded data on issues.

          Remember wine is not chasing a stationary target Microsoft and applications are always doing updates.

          duby229 you are saying the wine project should do something but you are not providing any method that it can. You are not allowing that wine project is running at resources limit almost continuously.
          I am not sure that is what he is saying? What i personally read from it, is that he is saying YOUR arguments for "over 6000 apps" is plainly wrong, and i agree.

          He is also saying "IF you are to CLAIM that "the winetest test 6000+ apps", you actually have to RUN THE APPS!".. And the winetest does not currently ACTUALLY run those apps, and as you point out - due to resource limits and time constraints.

          Winetests is what it is. Don't make it be something it is not.

          Do i have a better solution as it is now? Nope. Why is it always that if someone complains about something, all goes full on defensive and "Do it better yourself". Why not admit a spade is a spade? It is not perfect, and is not a real-world test.

          It should not be a "if things pass the test, it works" just as it sure does not mean "if it fails the test, it does not work", cos i can assure you that there is no "major flaw" with the windows installation i tested with, atleast not that "breaks" stuff in the way it happens when something does not work with wine. Yeah, M$ might have broken a whole lot of shit in their latest update, but i think its a slight stretch to semi-claim that the winetests are > M$ (even tho many of us Linux users hate M$ like the plague).

          AppDB site is highly unreliable, as there is extreme amounts of old useless information there (how did X game work with wine 1.2.2? Totally useless). Same with searching bugzilla for bugs, and you find 4-5-6-10 year old bugs related to antiquated wine versions? Wtf? Why? Yeah, it's a cost issue to run a website + server, but unless something better or more suited for feedbacks is made available, the feedback system will suck balls.
          The ONLY way to reliably be able to grab feedback from non-developers is to have a popup if you crash with a button to press. Type in info - mail - automagically attach the correct logs. You CANNOT expect Joe-Bob Average to figure out how to snip out the correct 2MB of the 25GB tracelog he has had after playing until he crashed that game/app now can you? The other non-dev posters that bewilder themselves into the bugzilla page soon give up cos they A)either repost already X year old bugs and get told off, or B)get told off right off the bat for not providing the correct logs. And then, if no noise is made = problem solved.
          Lot of the time the dev's have no way to test the actual REAL WORLD APPS (see reply above), and thus they CANNOT troubleshoot that particular issue. So how does SOMETIMES bugs get solved? Because a dev actually download/have bought/or otherwise are able to test the SAME app you have a bug with. There is no other way.
          There have been some posts from time to time with incentives to actually put together a pool of $ so devs could buy popular games and whatnot, but there has been NO word from any of the ppl in charge afaik.

          Sorry.. but that is the reality.

          Comment

          • oiaohm
            Senior Member
            • Mar 2017
            • 8391

            #35
            Originally posted by Cybmax View Post
            What i personally read from it, is that he is saying YOUR arguments for "over 6000 apps" is plainly wrong, and i agree.
            Problem is not plainly wrong. In the stable release stage happens every year real applications are run they are monitored for what they are using and compared against what the test suite is testing.

            Originally posted by Cybmax View Post
            He is also saying "IF you are to CLAIM that "the winetest test 6000+ apps", you actually have to RUN THE APPS!"..
            So the problem here is in one particular of the cycle what is in that wine testsuite is made from 6000+ applications. 6000+ applications may not include the application you are trying to run. Also being in that 6000 list does not mean wine can run that application properly.


            Originally posted by Cybmax View Post
            Why not admit a spade is a spade? It is not perfect, and is not a real-world test.
            Rember this the wine units tests based of real applications are in fact windows executable themselves.

            Originally posted by Cybmax View Post
            It should not be a "if things pass the test, it works" just as it sure does not mean "if it fails the test, it does not work", cos i can assure you that there is no "major flaw" with the windows installation i tested with, atleast not that "breaks" stuff in the way it happens when something does not work with wine. Yeah, M$ might have broken a whole lot of shit in their latest update, but i think its a slight stretch to semi-claim that the winetests are > M$ (even tho many of us Linux users hate M$ like the plague).
            You are ingoring the fact that the wine testsuite is built based off of behavours wine developers have seen applications require. So test fails in wine testsuite the application that that test was based off also fails. This is because that test suite has been built black box method recording what applications request of windows.

            Originally posted by Cybmax View Post
            AppDB site is highly unreliable, as there is extreme amounts of old useless information there (how did X game work with wine 1.2.2? Totally useless).
            That is lack of people power. For regression hunts even that old 1.2.2 test result helps to limit the search zone to find what broke a program. There would be less old information if we had people posting new test results. This is a pure lack of people power problem.

            Originally posted by Cybmax View Post
            Same with searching bugzilla for bugs, and you find 4-5-6-10 year old bugs related to antiquated wine versions? Wtf? Why?
            Simple answer lack of people power and most of those old bugs are still there and not fixed.

            Originally posted by Cybmax View Post
            Yeah, it's a cost issue to run a website + server, but unless something better or more suited for feedbacks is made available, the feedback system will suck balls. The ONLY way to reliably be able to grab feedback from non-developers is to have a popup if you crash with a button to press. Type in info - mail - automagically attach the correct logs.
            Yes there is a open feature request do this. So far no developer has come up with one that works.

            Originally posted by Cybmax View Post
            You CANNOT expect Joe-Bob Average to figure out how to snip out the correct 2MB of the 25GB tracelog he has had after playing until he crashed that game/app now can you?
            That is one of the problems. Even codeweavers with there commercial product how to extract that information. I learn the problem with a game called rollcage. I made 12 different patches to wine all wrong. Why because rollcage was crashing because the complete problem in the first 1kb of executable code had gone the wrong way. The game may have been appearing to run but it was running on a code path that should never run.

            The only way a Joe-Bob Average could possible snip out the correct 2MB of a 25GB tracelog without guide in automated way that has been come up with would be to run program on windows record a trace and then have the trace compared to on wine. Even this has a mega limitation.

            Basically if someone can come up with a tool todo this well the prototype automated reporting tool for wine could be activated. Basically this is the bug in the way of it.

            Originally posted by Cybmax View Post
            The other non-dev posters that bewilder themselves into the bugzilla page soon give up cos they A)either repost already X year old bugs and get told off, or B)get told off right off the bat for not providing the correct logs. And then, if no noise is made = problem solved.
            You are thinking we are not aware of this problem. You are not thinking we don't have the people power to fix it. Worse we don't have the software or design for software to fix it.


            Originally posted by Cybmax View Post
            Lot of the time the dev's have no way to test the actual REAL WORLD APPS (see reply above), and thus they CANNOT troubleshoot that particular issue.
            That is not what I said. Winetest suite makes sure 6000+ applications work. Developers working on a section of the api to make it work do get there hands on applications. Per development branch cycle they cannot test all 6000.

            Number of real applications tested per development branch cycle is less than 50 less than number of developers who worked on wine in that cycle. Please note I did not say that no real world applications were tested. The automated CI just cannot test them all.

            The wine developers are not forbin from running the applications to test their fixes. Heck they normally run applications create test-cases based on what those applications require add that to test-suite once their fix makes the test-suite work right then go back and run the real applicaiton and see if everything is really working right.

            At stable release mode where everything is restricted to bug fixing developers will test like 20+ applications each per day. That gives you less than 10 mins to look at an application and see if something is wrong. We need more people doing appdb reports where they can run the program for longer.

            Originally posted by Cybmax View Post
            So how does SOMETIMES bugs get solved? Because a dev actually download/have bought/or otherwise are able to test the SAME app you have a bug with. There is no other way.
            This is not how it always works. Rollcage game the developer never bought full version yes yet it got fixed. There was a demo the demo only showed one fault. That one fault was the code path error. After that was fixed the traces on the full version started making sense and he was able to fix the remaining faults without ever having the full game. But this took a person like me who was tolerant. It took 18 months to get rollcage to work with the developer putting stuff in reply to bug reports for me to test. So they either have to own the program or have a person willing to work with them testing stuff.


            Originally posted by Cybmax View Post
            There have been some posts from time to time with incentives to actually put together a pool of $ so devs could buy popular games and whatnot, but there has been NO word from any of the ppl in charge afaik.
            This idea is based on stupidity of not understanding the problem. Valve for a long time has allowed you buy game install it test if it works or not and return it for full refund. Even better they do extend to people who are proven wine developers to use games for free for testing reasons to fix bugs without needing to stop using the game. So all the popular games on steam are open to the wine developers now without needing to buy any. Problem here you still need the people power.

            So its not money to buy games is the problem. The problem is how to pay for more developers to work on wine there is more software accessible to wine developers than they can process.

            Yes the complete 11,118+ pool of steam is open to wine developers. + all the other free and trial software out there and this is without having to get money for the developers to buy software. Note that I said the testsuite of wine is only 6000+ applications. There is not enough people power to extend the testsuite out to cover all the steam applications yet let alone buying more. There is also not enough people power to test all those applications completely either.

            Wine project is so insanely short on people power its not funny.
            Last edited by oiaohm; 12 January 2020, 12:41 PM.

            Comment

            • Cybmax
              Phoronix Member
              • Jan 2019
              • 92

              #36
              Yeah. I think i actually get what you are saying. Yes, it "is" applications. I have also made an application.... however this is somewhat misleading.

              If i compile whatever "hello world" C sample, and run in with wine.. and submit this as an "app" to wine-devs proving that the console is able to output text.. would it then be 6001 apps in the tests? Cos this is NOT what i mean by saying "REAL WORLD APPS" tbh. But if this is what you mean, sure, there is 6000+ apps tested with wine. No problem.
              Custom made apps that does windows api things is not what i am thinking about when talking about this..

              If it is actually 6000+ applications (complete, with installers and whatnot), could you please list all of them, so i can go download a couple of them and test? I assume there is a list of apps since you know it is 6000+ of them yeah?

              Comment

              • oiaohm
                Senior Member
                • Mar 2017
                • 8391

                #37
                Originally posted by Cybmax View Post
                If i compile whatever "hello world" C sample, and run in with wine.. and submit this as an "app" to wine-devs proving that the console is able to output text.. would it then be 6001 apps in the tests? Cos this is NOT what i mean by saying "REAL WORLD APPS" tbh. But if this is what you mean, sure, there is 6000+ apps tested with wine. No problem. Custom made apps that does windows api things is not what i am thinking about when talking about this..
                That example of hello world c sample would unlikely increase the 6000+ applications you need to run to replicate what the winetest suite cover as to get in that list you need a unique behaviour of the windows API/ABI.

                Originally posted by Cybmax View Post
                If it is actually 6000+ applications (complete, with installers and whatnot), could you please list all of them, so i can go download a couple of them and test? I assume there is a list of apps since you know it is 6000+ of them yeah?
                There is a method to data-mine the list out the wine git while cross referring with the bugzilla, appdb result in giving you a list to replicate the wine testsuite. The bugzilla reports don't have a link to the appdb for no reason with the means to check who was the last to update that link . So the list is visible to those who know how to extract it so it public but hidden.

                I can tell you that 6000+ applications every one has a appdb entry. Also a mixture of platinum, gold, silver, bronze and garbage rated applications quite number of those that have not been tested on the appdb for many years. So the idea that is some basic program does not have grounds.

                The exact list use to built the testsuite is not give out random-ally for intentional reasons I will list.
                1) the wine project want to know the applications users are in fact using or at least trying to use. There is no point tell people what that list as does not get the information the project needs to focus the limited developer team the project has.

                2) Just because a application is in that list of 6000+ that the test suite covers does not mean it every version of that application. So the list itself could be highly miss leading on what will run and will not run. So you can see a person be a smart ass here pick up a few of them find that it don't work because their version is patched different and then incorrectly claim wine testsuite does not work when it working fine.

                3) Finally just because a application has unique tests in the wine testsuite does not mean that application fully works/fully runs yet. Sometimes the reason why application is in the testsuite list is that it does not work yet and when those test pass retest application and then maybe add more tests.

                Yes the testsuite report of 356 todos on the top page gets even bigger as you go in is that todos is something the wine code code base knows that wine should do right but it currently does not that in fact breaks behaviour application(as in application runs but does not run right). Yes doing a simple search on the wine source fo the TODO entries give a nice massive number.

                When wine testsuite is running on windows we do not have the todo information we just have the absolutely fatal information. Windows failing the testsuite is just like wine failing the testsuite this is basically telling you applications that don't work.

                Remember what you did your complained about appdb data being old. Pick the applications you have interest in that don't have current reports and update the appdb status this will give the project more of the information it needs.

                Read back carefully I did not say that 6000+ the testsuite is testing fully worked its the number of applications you would need to run to replicate the wine testsuite coverage.

                See problem here you total arguement has been based on presumes. Some of those presumes have result in people not updating appdb. Like program don't work no point doing appdb entry right? Wrong doing appdb entry shows interest in having that application work. Application don't work don't do a test report right? Wrong again doing the test report be it the application work or not work shows interest in wanting the application work.

                Wine project really works from the squeaking wheel gets grease the non squeaking wheel that is just as broken does not.

                Comment

                • Cybmax
                  Phoronix Member
                  • Jan 2019
                  • 92

                  #38
                  I am not entirely sure i understand the concept of "Test Suite".
                  I will try to ask some questions to better understand this.
                  1. Is "Wine test suite" the automated test program everyone can download and run? (ie. winetest-latest.exe)
                  2. Is "Wine test suite" all the reports added on the appdb webpage? (eg. https://appdb.winehq.org/objectManag...sion&iId=25823 being one?)

                  My understanding of the 2 questions is:
                  1. winetest-latest.exe runs a series of api functions, and report if they work or not - and is basically 1 app doing tests, much like eg. a d3d benchmark program would do.
                  2. Users are reporting how the apps work on appdb, and tweaks/hacks/installation methods to get the app to work. This is not tested "by the devs", as this is a massive list of various apps.. some works, some dont, and as you say some versions have "Platinum" rating, vs a different version can have a different rating.

                  If a "Garbage" rating is given for #2, there is no automation of it being a bug report generated on bugzilla, and my understanding is that those 2 systems - appdb <-> bugzilla is two different beasts.

                  Comment

                  • oiaohm
                    Senior Member
                    • Mar 2017
                    • 8391

                    #39
                    Originally posted by Cybmax View Post
                    1. winetest-latest.exe runs a series of api functions, and report if they work or not - and is basically 1 app doing tests, much like eg. a d3d benchmark program would do.
                    Winetest-latest,exe is not in fact 1 application.
                    https://github.com/wine-mirror/wine/...grams/winetest its a front end.

                    Each dll has it own conformance test that is it own program that is bundled into winetest-latest.exe this is why you can have a crash test and not lose results in a winetest run. So winetest is really a logging solution for a collection of applications that make up the testsuite. Those applications are not counted to the 6000 applications required to replicate it testing.

                    Originally posted by Cybmax View Post
                    2. Users are reporting how the apps work on appdb, and tweaks/hacks/installation methods to get the app to work.
                    True to this point.
                    Originally posted by Cybmax View Post
                    This is not tested "by the devs",
                    This is not 100% true. Developers of wine are allowed to be users of application as well so they can be/do writing the reports on the appdb and the test reports at times. So particular entries in the appdb are written by developers of wine. Appdb is a mixture of non developer end user reports and wine developer end user reports.

                    Originally posted by Cybmax View Post
                    as this is a massive list of various apps.. some works, some dont,
                    Basically there is not enough labour for the developers of wine to-do all the real world application testing.


                    Originally posted by Cybmax View Post
                    If a "Garbage" rating is given for #2, there is no automation of it being a bug report generated on bugzilla,
                    True a garbage test result in appdb does not automatically generate a bugzilla entry in the same way a failure of the winetest suite does not generate a bugzilla entry. Same problem no one made a system that can automated generate sanity dependable enough. Valve is with the protondb is experimenting with a few things that may come back to wine appdb/bugzilla in time but they are still having a 60% screw up rate.



                    Even that Garbage rating does not generate a bug report on bugzilla if the Maintainer of the application appdb page is doing their job(yes if we have enough labour) they should review the test Garbage results and file bugzilla entries linked to appdb entry as required.

                    Originally posted by Cybmax View Post
                    my understanding is that those 2 systems - appdb <-> bugzilla is two different beasts.
                    By wine project policies they are not two different beasts. If the issues with protondb can be worked out they may not remain two different beasts. They are currently two different beasts with a policy for a human to follow in the middle to join them into 1 because that the best tech we have at this time.

                    Originally posted by Cybmax View Post
                    Is "Wine test suite" all the reports added on the appdb webpage?
                    The answer is no wine test suite results are strictly sent to test.winehq.org.

                    The wine test suite tests are designed from a very particular version of an application so application gets a update the wine test suite could say that application was still fine when it was not or the reverse that application does not work when it does. So putting the information straight from the wine testsuite on the appdb page would not be wise.

                    The google developer who started winetricks did attempt a automated test suite using real world applications and it turned out to be just as defective as the wine test suite unit tests for correct appdb information and worse gave no clear understand what the heck was going on when X application worked on Y version number of windows/wine but did not work on Z version number of windows/wine. So end up being practically useless. So its not guess work that duby229 cannot work and is a pointless route because the route as been tried and failed spectacularly.

                    Yes this is how testing with real world applications in automated suite would backfire for appdb data is the same way the wine test suite backfires for appdb data so neither can be used to fill in appdb data.

                    So we need humans doing the testing and reporting who are smart enough to read applications notices like hey this program has a update install it. Yes AI are not that bright yet. The appdb is the best system that the wine project has been able to design for this at this stage(this is not that the project is not looking for better).

                    Monitoring the appdb page for reports of new failures inform developers when the testsuite need a rework due to application changes. Now if that information is not being put on the appdb then we have a problem.

                    Wine development process is quite a complex integrated system. Not as automated as we would like and really need more human labour to work as designed.

                    The big thing wine project is massively short of is human labour. Remember Microsoft can throw over 2000 developers full time at Windows changing things wine project has less than 50 even counting part timers. Also Microsoft can throw 4000+ personal at release testing also paid full time.

                    The wine project is critically under resourced.

                    Comment

                    • duby229
                      Senior Member
                      • Nov 2007
                      • 7779

                      #40
                      Originally posted by oiaohm View Post
                      Winetest-latest,exe is not in fact 1 application.
                      https://github.com/wine-mirror/wine/...grams/winetest its a front end.

                      Each dll has it own conformance test that is it own program that is bundled into winetest-latest.exe this is why you can have a crash test and not lose results in a winetest run. So winetest is really a logging solution for a collection of applications that make up the testsuite. Those applications are not counted to the 6000 applications required to replicate it testing.



                      True to this point.

                      This is not 100% true. Developers of wine are allowed to be users of application as well so they can be/do writing the reports on the appdb and the test reports at times. So particular entries in the appdb are written by developers of wine. Appdb is a mixture of non developer end user reports and wine developer end user reports.


                      Basically there is not enough labour for the developers of wine to-do all the real world application testing.



                      True a garbage test result in appdb does not automatically generate a bugzilla entry in the same way a failure of the winetest suite does not generate a bugzilla entry. Same problem no one made a system that can automated generate sanity dependable enough. Valve is with the protondb is experimenting with a few things that may come back to wine appdb/bugzilla in time but they are still having a 60% screw up rate.



                      Even that Garbage rating does not generate a bug report on bugzilla if the Maintainer of the application appdb page is doing their job(yes if we have enough labour) they should review the test Garbage results and file bugzilla entries linked to appdb entry as required.



                      By wine project policies they are not two different beasts. If the issues with protondb can be worked out they may not remain two different beasts. They are currently two different beasts with a policy for a human to follow in the middle to join them into 1 because that the best tech we have at this time.



                      The answer is no wine test suite results are strictly sent to test.winehq.org.

                      The wine test suite tests are designed from a very particular version of an application so application gets a update the wine test suite could say that application was still fine when it was not or the reverse that application does not work when it does. So putting the information straight from the wine testsuite on the appdb page would not be wise.

                      The google developer who started winetricks did attempt a automated test suite using real world applications and it turned out to be just as defective as the wine test suite unit tests for correct appdb information and worse gave no clear understand what the heck was going on when X application worked on Y version number of windows/wine but did not work on Z version number of windows/wine. So end up being practically useless. So its not guess work that duby229 cannot work and is a pointless route because the route as been tried and failed spectacularly.

                      Yes this is how testing with real world applications in automated suite would backfire for appdb data is the same way the wine test suite backfires for appdb data so neither can be used to fill in appdb data.

                      So we need humans doing the testing and reporting who are smart enough to read applications notices like hey this program has a update install it. Yes AI are not that bright yet. The appdb is the best system that the wine project has been able to design for this at this stage(this is not that the project is not looking for better).

                      Monitoring the appdb page for reports of new failures inform developers when the testsuite need a rework due to application changes. Now if that information is not being put on the appdb then we have a problem.

                      Wine development process is quite a complex integrated system. Not as automated as we would like and really need more human labour to work as designed.

                      The big thing wine project is massively short of is human labour. Remember Microsoft can throw over 2000 developers full time at Windows changing things wine project has less than 50 even counting part timers. Also Microsoft can throw 4000+ personal at release testing also paid full time.

                      The wine project is critically under resourced.
                      Another wall of nonsense...

                      Facts are what facts are, bottom line.... Wine is what it is and time itself has already proven what its doing doesn't work. EVERY single releases regresses badly... Wine unit tests are imaginary at best and wine devs refuse to emulate windows behavior and so long as that's true wine will continue to regress badly every single release.

                      Comment

                      Working...
                      X