Running Benchmarks On Other New Linux Games

If in recent Linux graphics card reviews and driver articles you have been wanting to see other Linux games benchmarked, you should definitely read this article.

With more and more games appearing on Linux, it seems each time a GPU hardware or driver article is published on Phoronix there are requests to see other games/benchmarks included... It's usually very repetitive but the same questions/requests keep coming up. Besides Source Engine games being more CPU limited than GPU limited these days, I've already clearly laid out in the past why I don't like benchmarking with Steam games: the payloads are usually quite large, Steam runs automatic updates to make being able to bind a test to a given version for reproducibility/comparisons harder, and it's a harder process all around to automate.

To reiterate some points from that aforelinked article and other statements in the past, my Linux gaming benchmark requirements usually come down to:

- The most important one is that the game/application must be able to run an automated test case / time demo / replay in a fully-automated manner. It must be script-able to function. I will not run tests that can't be fully-automated. It's more accurate if it's 100% automated, it's easier for others to reproduce the tests on their own hardware/software, and it saves me a heck of a lot of time. I single-handedly write hundreds of articles per month and carry out a countless number of benchmarks each month and will not run any tests if they aren't automated.

- I have no objections to benchmarking closed-source games. Obviously it's nicer benchmarking open-source software, but I have no issues -- nor for running PTS tests -- if it is of closed-source software as long as it makes a good test.

- If the game is retail-only I have no real objections either: the Phoronix Test Suite has test profiles for Doom 3, Quake 4, Counter-Strike: Source, and other pay-to-play titles. But of course for broad adoption and widespread use of the games as a benchmark, it's convenient to see the games free. If there's a demo of the game that's also better since it usually means a smaller download for making a benchmark.

If the benchmark can be automated fully and scripted, then it's very easy for me to incorporate it into the Phoronix Test Suite. Besides being my own software, Phoronix Test Suite profiles are useful beyond being automated because:

- All the commands and processes are fully standardized. There's also uniform graphs and automated hardware/software reporting.

- Numerous features like measuring standard deviation and automatically increasing the number of times to run a benchmark if a threshold is exceeded, automatic suspending of screensavers for benchmarks, logging of system data, supporting sensor monitoring when running tests, etc.

- By having a Phoronix Test Suite test profile of the game it's generally then pulled in by a wide range of organization and companies that rely upon PTS for facilitating their internal QA and performance tests. Numerous companies will then automatically begin running benchmarks with the said game/application.

- Tests can be shared and compared via the unique OpenBenchmarking.org platform.

If the workload can be automated, it's very easy to write a test profile. For a look at raw test profiles look inside (and feel free to modify) ~/.phoronix-test-suite/test-profiles/pts/. Here's some examples:

- Xonotic is a complete but a more complex test profile example that takes full advantage of all PTS features, works on not only Linux but also OS X and Windows, and automates everything from downloading of files to handling of the graphics settings.

- Team Fortress 2 for a Steam game example on Linux.

- C-Ray for a simple non-game test profile example.

- And the hundreds of other tests.

So with all of that said if there's some new game or other application you want to see benchmarked on Phoronix, here's the following recommendations:

- The most ideal path is if you -- whether you're an independent user or the creator of the software itself -- is to write your own test profile. There's many benefits in doing so and will ensure many users of the said game/software for testing purposes. Even if you don't make the test profile yourself, if you can send over a simple script that yields an example of an automated run, from there it's very easy for me to turn it into a proper Phoronix Test Suite test profile. Feel free to contact me for more information on getting involved or how to make a test profile; it's actually quite easy but admittedly there's some documentation lacking so feel free to contact me.

- If you aren't going to submit a test case yourself but are just making a request, first do your due diligence. For example, in recent weeks I've had countless requests of having Metro: Last Light benchmarks on Linux... And I've had to respond as many times -- via the forums, Twitter, etc -- that the Linux version of Metro: Last Light in Steam doesn't even have the benchmarking mode that's found on Windows. So make sure the game can at least be fully-automated and suits my requirements before trying to get me to run said test(s).

- If you really would like a test to be used and aren't willing to make it yourself, hopefully you subscribe to Phoronix Premium or make your recommendation with a PayPal contribution. I work seven days per week, about 100 hours per week, and am writing hundreds of Phoronix articles per month beyond my other job roles and PTS workloads. If you're making a request, it's higher on my TODO list if I'm motivated to try to find the time to explore such a test.

- For companies and organizations seeking custom test development and other custom engineering services, there are commercial PTS opportunities.