Announcement

Collapse
No announcement yet.

OpenRA test suite

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • OpenRA test suite

    OpenRA is an Open Source recreation of the classic Westwood Studios real-time strategy games such as Command & Conquer: Red Alert, Tiberian Dawn and Dune 2000. The old-school graphics often make people believe this is the same game that ran just fine on their Pentium II in the 90s. In fact it is a modern recreation designed for current hardware. It runs in any screen resolution and has no hard-coded limits of the legacy games such as map size or AI search radius which certainly improves it, but also makes it more demanding.

    Those games look simple, but are in fact complex simulations with a lot of variables. Performance is an on-going topic in development with constant improvements, but also set-backs when new features appear or it is sacrificed for easier maintenance. The bug tracker has random entries about people complaining the game feels sluggish. These are often not reproducible as basic information and hard data is missing.

    Long story short: a Phoronix Test Suite benchmark might be interesting especially as I don't see any RTS game in the collection. What is required to create one? Which output format for CPU and rendering eases parsing and analysis?
    Open Source reimplementation of Westwood Studios' 2D Command and Conquer games

  • #2
    If you can write a bash script to launch the game automatically, run whatever reproducible test in the game, and to then exit the game while dumping out the frame-rate or whatever, from there it is trivial to create the test profile part.
    Michael Larabel
    http://www.michaellarabel.com/

    Comment


    • #3
      I opened in ticket in our bug tracker about it. We can run both replays and scripted maps as well as hide the GUI. Do most games dump out an averaged frame-rate in the end? What is required as command line parameter except enable benchmark mode? Something like a runtime in seconds when to automatically exit?
      OpenRA - Open Source real-time strategy game engine for early Westwood games such as Command & Conquer: Red Alert written in C# using SDL and OpenGL. Runs on Windows, Linux, *BSD and Mac OS X.

      Comment


      • #4
        Originally posted by Mailaender View Post
        I opened in ticket in our bug tracker about it. We can run both replays and scripted maps as well as hide the GUI. Do most games dump out an averaged frame-rate in the end? What is required as command line parameter except enable benchmark mode? Something like a runtime in seconds when to automatically exit?
        Sorry for the late reply, somehow missed it. Yes, most dump out the average FPS (and ideally the min/max FPS).

        From CLI would be nice to enable benchmark, set resolution, and any major graphical quality settings. If say --benchmark automatically starts up the test and then exits while dumping the results either to stdout or some file.
        Michael Larabel
        http://www.michaellarabel.com/

        Comment


        • #5
          Is https://github.com/OpenRA/OpenRA/pull/10133 something that could work? Settings including screen resolutions and FPS caps are already available via the command line and configuration files.
          Last edited by Mailaender; 11-28-2015, 09:25 AM.

          Comment


          • #6
            Originally posted by Mailaender View Post
            Is https://github.com/OpenRA/OpenRA/pull/10133 something that could work?
            Looks like the benchmark mode would work and the CSV output. Is there also a way to specify the resolution (and if relevant any like low/medium/high image quality options)?
            Michael Larabel
            http://www.michaellarabel.com/

            Comment


            • #7
              There currently is no way of turning of effects or otherwise changing rendering quality. However you can switch resolutions or un-cap the frame rate at game start. I will update the auto-generated man page with documentation on relevant parameters as well.

              Comment


              • #8
                I placed the following files into ~/.phoronix-test-suite/test-profiles/local/openra-20151229/ but sadly it won't download and run anything. Note: I modified existing test profiles as I couldn't find any recent documentation on it. Don't really know what I am doing especially what values are allowed and mandatory in these configuration files. ResultScale and Proportion are probably wrong as well as we are not doing FPS, but render/tick times (the lower the better) here. If the benchmark runs, there probably has to be a way to stop it otherwise the game will run indefinitely.

                downloads.xml:
                HTML Code:
                <?xml version="1.0"?>
                <PhoronixTestSuite>
                  <Downloads>
                    <Package>
                      <URL>https://github.com/OpenRA/OpenRA/archive/e38424a2eb7e3562b4e2f0eaa2c58006e4f11b62.tar.gz<URL>
                      <MD5>e82e5869da534a330228c5e2b90f7727</MD5>
                    </Package>
                  </Downloads>
                </PhoronixTestSuite>
                install.sh:
                Code:
                #!/bin/sh
                
                tar -xvf *.tar.gz
                
                cd OpenRA-*
                
                make all
                
                echo "#!/bin/sh
                mono OpenRA.Game.exe" > openra
                
                chmod +x openra
                post.sh:
                Code:
                cat ~/.openra/Logs/cpu.csv | awk -F';' '{sum+=$2; ++n} END { print "CPU average: "sum/n" ms" }'
                cat ~/.openra/Logs/render.csv | awk -F';' '{sum+=$2; ++n} END { print "GPU average: "sum/n" ms" }'
                results-definition.xml
                HTML Code:
                <?xml version="1.0"?>
                <PhoronixTestSuite>
                  <ResultsParser>
                    <OutputTemplate>CPU average: #_RESULT_# ms</OutputTemplate>
                    <LineHint>CPU</LineHint>
                  </ResultsParser>
                  <ResultsParser>
                    <OutputTemplate>GPU average: #_RESULT_# ms</OutputTemplate>
                    <LineHint>GPU</LineHint>
                  </ResultsParser>
                </PhoronixTestSuite>
                test-definition.xml
                HTML Code:
                <?xml version="1.0"?>
                <PhoronixTestSuite>
                  <TestInformation>
                    <Title>OpenRA</Title>
                    <AppVersion>20151229</AppVersion>
                    <Description>Open Source real-time strategy game engine for early Westwood games such as Command &amp; Conquer: Red Alert written in C# using SDL and OpenGL.</Description>
                    <ResultScale>MiliSecond</ResultScale>
                    <Proportion>HIB</Proportion>
                    <TimesToRun>1</TimesToRun>
                  </TestInformation>
                  <TestProfile>
                    <Version>20151229</Version>
                    <SupportedPlatforms>Linux</SupportedPlatforms>
                    <SoftwareType>Game</SoftwareType>
                    <TestType>Graphics</TestType>
                    <License>Free</License>
                    <SupportedArchitectures>x86, x86_64</SupportedArchitectures>
                    <ExternalDependencies>mono</ExternalDependencies>
                    <EnvironmentSize>100</EnvironmentSize>
                    <ProjectURL>http://www.openra.net/</ProjectURL>
                    <Maintainer>Matthias Mailänder</Maintainer>
                  </TestProfile>
                  <TestSettings>
                    <Default>
                      <Arguments>Launch.Benchmark=true</Arguments>
                    </Default>
                  </TestSettings>
                </PhoronixTestSuite>
                Also https://openbenchmarking.org/create_suite isn't available and phoronix-test-suite upload-test-suite openra-20151229 gives me the following error:

                Phoronix Test Suite v6.0.1
                openra-20151229


                [NOTICE] Undefined: pts_test_profile::$xml_parser in pts_validation:104

                Fatal error: Call to a member function getFileLocation() on null in /usr/share/phoronix-test-suite/pts-core/objects/pts_validation.php on line 104

                Comment


                • #9
                  Originally posted by Mailaender View Post
                  I placed the following files into ~/.phoronix-test-suite/test-profiles/local/openra-20151229/ but sadly it won't download and run anything. Note: I modified existing test profiles as I couldn't find any recent documentation on it. Don't really know what I am doing especially what values are allowed and mandatory in these configuration files. ResultScale and Proportion are probably wrong as well as we are not doing FPS, but render/tick times (the lower the better) here. If the benchmark runs, there probably has to be a way to stop it otherwise the game will run indefinitely.

                  downloads.xml:
                  HTML Code:
                  <?xml version="1.0"?>
                  <PhoronixTestSuite>
                  <Downloads>
                  <Package>
                  <URL>https://github.com/OpenRA/OpenRA/archive/e38424a2eb7e3562b4e2f0eaa2c58006e4f11b62.tar.gz<URL>
                  <MD5>e82e5869da534a330228c5e2b90f7727</MD5>
                  </Package>
                  </Downloads>
                  </PhoronixTestSuite>
                  install.sh:
                  Code:
                  #!/bin/sh
                  
                  tar -xvf *.tar.gz
                  
                  cd OpenRA-*
                  
                  make all
                  
                  echo "#!/bin/sh
                  mono OpenRA.Game.exe" > openra
                  
                  chmod +x openra
                  post.sh:
                  Code:
                  cat ~/.openra/Logs/cpu.csv | awk -F';' '{sum+=$2; ++n} END { print "CPU average: "sum/n" ms" }'
                  cat ~/.openra/Logs/render.csv | awk -F';' '{sum+=$2; ++n} END { print "GPU average: "sum/n" ms" }'
                  results-definition.xml
                  HTML Code:
                  <?xml version="1.0"?>
                  <PhoronixTestSuite>
                  <ResultsParser>
                  <OutputTemplate>CPU average: #_RESULT_# ms</OutputTemplate>
                  <LineHint>CPU</LineHint>
                  </ResultsParser>
                  <ResultsParser>
                  <OutputTemplate>GPU average: #_RESULT_# ms</OutputTemplate>
                  <LineHint>GPU</LineHint>
                  </ResultsParser>
                  </PhoronixTestSuite>
                  test-definition.xml
                  HTML Code:
                  <?xml version="1.0"?>
                  <PhoronixTestSuite>
                  <TestInformation>
                  <Title>OpenRA</Title>
                  <AppVersion>20151229</AppVersion>
                  <Description>Open Source real-time strategy game engine for early Westwood games such as Command &amp; Conquer: Red Alert written in C# using SDL and OpenGL.</Description>
                  <ResultScale>MiliSecond</ResultScale>
                  <Proportion>HIB</Proportion>
                  <TimesToRun>1</TimesToRun>
                  </TestInformation>
                  <TestProfile>
                  <Version>20151229</Version>
                  <SupportedPlatforms>Linux</SupportedPlatforms>
                  <SoftwareType>Game</SoftwareType>
                  <TestType>Graphics</TestType>
                  <License>Free</License>
                  <SupportedArchitectures>x86, x86_64</SupportedArchitectures>
                  <ExternalDependencies>mono</ExternalDependencies>
                  <EnvironmentSize>100</EnvironmentSize>
                  <ProjectURL>http://www.openra.net/</ProjectURL>
                  <Maintainer>Matthias Mailänder</Maintainer>
                  </TestProfile>
                  <TestSettings>
                  <Default>
                  <Arguments>Launch.Benchmark=true</Arguments>
                  </Default>
                  </TestSettings>
                  </PhoronixTestSuite>
                  Also https://openbenchmarking.org/create_suite isn't available and phoronix-test-suite upload-test-suite openra-20151229 gives me the following error:
                  It would be upload-test-profile, not upload-test-suite. It's a test profile you are making. What if you rename that folder to just openra and then do phoronix-test-suite benchmark openra ?

                  Yes, you would also need a way to allow the game to stop running when desired. Does that launch benchmark mode not stop after a defined time or anything?
                  Michael Larabel
                  http://www.michaellarabel.com/

                  Comment


                  • #10
                    Ah nice upload-test-profile also gives some XML validation. Looks like my downloads.xml was not conforming somehow which I was able to fix. Still the download fails and I am not sure why. Got the files validated, but the result seems to crash the server anyway: https://openbenchmarking.org/test/matt/openra
                    Last edited by Mailaender; 01-02-2016, 01:20 PM.

                    Comment

                    Working...
                    X