OpenRA test suite

[Mailaender]

OpenRA is an Open Source recreation of the classic Westwood Studios real-time strategy games such as Command & Conquer: Red Alert, Tiberian Dawn and Dune 2000. The old-school graphics often make people believe this is the same game that ran just fine on their Pentium II in the 90s. In fact it is a modern recreation designed for current hardware. It runs in any screen resolution and has no hard-coded limits of the legacy games such as map size or AI search radius which certainly improves it, but also makes it more demanding.

Those games look simple, but are in fact complex simulations with a lot of variables. Performance is an on-going topic in development with constant improvements, but also set-backs when new features appear or it is sacrificed for easier maintenance. The bug tracker has random entries about people complaining the game feels sluggish. These are often not reproducible as basic information and hard data is missing.

Long story short: a Phoronix Test Suite benchmark might be interesting especially as I don't see any RTS game in the collection. What is required to create one? Which output format for CPU and rendering eases parsing and analysis?

[Michael]

If you can write a bash script to launch the game automatically, run whatever reproducible test in the game, and to then exit the game while dumping out the frame-rate or whatever, from there it is trivial to create the test profile part.

[Mailaender]

I opened in ticket in our bug tracker about it. We can run both replays and scripted maps as well as hide the GUI. Do most games dump out an averaged frame-rate in the end? What is required as command line parameter except enable benchmark mode? Something like a runtime in seconds when to automatically exit?

[Michael]

I opened in ticket in our bug tracker about it. We can run both replays and scripted maps as well as hide the GUI. Do most games dump out an averaged frame-rate in the end? What is required as command line parameter except enable benchmark mode? Something like a runtime in seconds when to automatically exit?

Sorry for the late reply, somehow missed it. Yes, most dump out the average FPS (and ideally the min/max FPS).

From CLI would be nice to enable benchmark, set resolution, and any major graphical quality settings. If say --benchmark automatically starts up the test and then exits while dumping the results either to stdout or some file.

[Mailaender]

Is https://github.com/OpenRA/OpenRA/pull/10133 something that could work? Settings including screen resolutions and FPS caps are already available via the command line and configuration files.

[Michael]

Is https://github.com/OpenRA/OpenRA/pull/10133 something that could work?

Looks like the benchmark mode would work and the CSV output. Is there also a way to specify the resolution (and if relevant any like low/medium/high image quality options)?

[Mailaender]

There currently is no way of turning of effects or otherwise changing rendering quality. However you can switch resolutions or un-cap the frame rate at game start. I will update the auto-generated man page with documentation on relevant parameters as well.

[Mailaender]

I placed the following files into ~/.phoronix-test-suite/test-profiles/local/openra-20151229/ but sadly it won't download and run anything. Note: I modified existing test profiles as I couldn't find any recent documentation on it. Don't really know what I am doing especially what values are allowed and mandatory in these configuration files. ResultScale and Proportion are probably wrong as well as we are not doing FPS, but render/tick times (the lower the better) here. If the benchmark runs, there probably has to be a way to stop it otherwise the game will run indefinitely.

downloads.xml:

<?xml version="1.0"?>
<PhoronixTestSuite>
  <Downloads>
    <Package>
      <URL>https://github.com/OpenRA/OpenRA/archive/e38424a2eb7e3562b4e2f0eaa2c58006e4f11b62.tar.gz<URL>
      <MD5>e82e5869da534a330228c5e2b90f7727</MD5>
    </Package>
  </Downloads>
</PhoronixTestSuite>

install.sh:

#!/bin/sh

tar -xvf *.tar.gz

cd OpenRA-*

make all

echo "#!/bin/sh
mono OpenRA.Game.exe" > openra

chmod +x openra

post.sh:

cat ~/.openra/Logs/cpu.csv | awk -F';' '{sum+=$2; ++n} END { print "CPU average: "sum/n" ms" }'
cat ~/.openra/Logs/render.csv | awk -F';' '{sum+=$2; ++n} END { print "GPU average: "sum/n" ms" }'

results-definition.xml

<?xml version="1.0"?>
<PhoronixTestSuite>
  <ResultsParser>
    <OutputTemplate>CPU average: #_RESULT_# ms</OutputTemplate>
    <LineHint>CPU</LineHint>
  </ResultsParser>
  <ResultsParser>
    <OutputTemplate>GPU average: #_RESULT_# ms</OutputTemplate>
    <LineHint>GPU</LineHint>
  </ResultsParser>
</PhoronixTestSuite>

test-definition.xml

<?xml version="1.0"?>
<PhoronixTestSuite>
  <TestInformation>
    <Title>OpenRA</Title>
    <AppVersion>20151229</AppVersion>
    <Description>Open Source real-time strategy game engine for early Westwood games such as Command &amp; Conquer: Red Alert written in C# using SDL and OpenGL.</Description>
    <ResultScale>MiliSecond</ResultScale>
    <Proportion>HIB</Proportion>
    <TimesToRun>1</TimesToRun>
  </TestInformation>
  <TestProfile>
    <Version>20151229</Version>
    <SupportedPlatforms>Linux</SupportedPlatforms>
    <SoftwareType>Game</SoftwareType>
    <TestType>Graphics</TestType>
    <License>Free</License>
    <SupportedArchitectures>x86, x86_64</SupportedArchitectures>
    <ExternalDependencies>mono</ExternalDependencies>
    <EnvironmentSize>100</EnvironmentSize>
    <ProjectURL>http://www.openra.net/</ProjectURL>
    <Maintainer>Matthias Mailänder</Maintainer>
  </TestProfile>
  <TestSettings>
    <Default>
      <Arguments>Launch.Benchmark=true</Arguments>
    </Default>
  </TestSettings>
</PhoronixTestSuite>

Also https://openbenchmarking.org/create_suite isn't available and phoronix-test-suite upload-test-suite openra-20151229 gives me the following error:

Phoronix Test Suite v6.0.1
openra-20151229


[NOTICE] Undefined: pts_test_profile::$xml_parser in pts_validation:104

Fatal error: Call to a member function getFileLocation() on null in /usr/share/phoronix-test-suite/pts-core/objects/pts_validation.php on line 104

[Michael]

I placed the following files into ~/.phoronix-test-suite/test-profiles/local/openra-20151229/ but sadly it won't download and run anything. Note: I modified existing test profiles as I couldn't find any recent documentation on it. Don't really know what I am doing especially what values are allowed and mandatory in these configuration files. ResultScale and Proportion are probably wrong as well as we are not doing FPS, but render/tick times (the lower the better) here. If the benchmark runs, there probably has to be a way to stop it otherwise the game will run indefinitely.

downloads.xml:

<?xml version="1.0"?>
<PhoronixTestSuite>
<Downloads>
<Package>
<URL>https://github.com/OpenRA/OpenRA/archive/e38424a2eb7e3562b4e2f0eaa2c58006e4f11b62.tar.gz<URL>
<MD5>e82e5869da534a330228c5e2b90f7727</MD5>
</Package>
</Downloads>
</PhoronixTestSuite>

install.sh:

#!/bin/sh

tar -xvf *.tar.gz

cd OpenRA-*

make all

echo "#!/bin/sh
mono OpenRA.Game.exe" > openra

chmod +x openra

post.sh:

cat ~/.openra/Logs/cpu.csv | awk -F';' '{sum+=$2; ++n} END { print "CPU average: "sum/n" ms" }'
cat ~/.openra/Logs/render.csv | awk -F';' '{sum+=$2; ++n} END { print "GPU average: "sum/n" ms" }'

results-definition.xml

<?xml version="1.0"?>
<PhoronixTestSuite>
<ResultsParser>
<OutputTemplate>CPU average: #_RESULT_# ms</OutputTemplate>
<LineHint>CPU</LineHint>
</ResultsParser>
<ResultsParser>
<OutputTemplate>GPU average: #_RESULT_# ms</OutputTemplate>
<LineHint>GPU</LineHint>
</ResultsParser>
</PhoronixTestSuite>

test-definition.xml

<?xml version="1.0"?>
<PhoronixTestSuite>
<TestInformation>
<Title>OpenRA</Title>
<AppVersion>20151229</AppVersion>
<Description>Open Source real-time strategy game engine for early Westwood games such as Command &amp; Conquer: Red Alert written in C# using SDL and OpenGL.</Description>
<ResultScale>MiliSecond</ResultScale>
<Proportion>HIB</Proportion>
<TimesToRun>1</TimesToRun>
</TestInformation>
<TestProfile>
<Version>20151229</Version>
<SupportedPlatforms>Linux</SupportedPlatforms>
<SoftwareType>Game</SoftwareType>
<TestType>Graphics</TestType>
<License>Free</License>
<SupportedArchitectures>x86, x86_64</SupportedArchitectures>
<ExternalDependencies>mono</ExternalDependencies>
<EnvironmentSize>100</EnvironmentSize>
<ProjectURL>http://www.openra.net/</ProjectURL>
<Maintainer>Matthias Mailänder</Maintainer>
</TestProfile>
<TestSettings>
<Default>
<Arguments>Launch.Benchmark=true</Arguments>
</Default>
</TestSettings>
</PhoronixTestSuite>

Also https://openbenchmarking.org/create_suite isn't available and phoronix-test-suite upload-test-suite openra-20151229 gives me the following error:

It would be upload-test-profile, not upload-test-suite. It's a test profile you are making. What if you rename that folder to just openra and then do phoronix-test-suite benchmark openra ?

Yes, you would also need a way to allow the game to stop running when desired. Does that launch benchmark mode not stop after a defined time or anything?

[Mailaender]

Ah nice upload-test-profile also gives some XML validation. Looks like my downloads.xml was not conforming somehow which I was able to fix. Still the download fails and I am not sure why. Got the files validated, but the result seems to crash the server anyway: https://openbenchmarking.org/test/matt/openra