1. Computers
  2. Display Drivers
  3. Graphics Cards
  4. Memory
  5. Motherboards
  6. Processors
  7. Software
  8. Storage
  9. Operating Systems


Facebook RSS Twitter Twitter Google Plus


Phoronix Test Suite

OpenBenchmarking.org

Running Benchmarks On Other New Linux Games

Phoronix

Published on 25 November 2013 05:13 AM EST
Written by Michael Larabel in Phoronix
48 Comments

If in recent Linux graphics card reviews and driver articles you have been wanting to see other Linux games benchmarked, you should definitely read this article.

With more and more games appearing on Linux, it seems each time a GPU hardware or driver article is published on Phoronix there are requests to see other games/benchmarks included... It's usually very repetitive but the same questions/requests keep coming up. Besides Source Engine games being more CPU limited than GPU limited these days, I've already clearly laid out in the past why I don't like benchmarking with Steam games: the payloads are usually quite large, Steam runs automatic updates to make being able to bind a test to a given version for reproducibility/comparisons harder, and it's a harder process all around to automate.

To reiterate some points from that aforelinked article and other statements in the past, my Linux gaming benchmark requirements usually come down to:

- The most important one is that the game/application must be able to run an automated test case / time demo / replay in a fully-automated manner. It must be script-able to function. I will not run tests that can't be fully-automated. It's more accurate if it's 100% automated, it's easier for others to reproduce the tests on their own hardware/software, and it saves me a heck of a lot of time. I single-handedly write hundreds of articles per month and carry out a countless number of benchmarks each month and will not run any tests if they aren't automated.

- I have no objections to benchmarking closed-source games. Obviously it's nicer benchmarking open-source software, but I have no issues -- nor for running PTS tests -- if it is of closed-source software as long as it makes a good test.

- If the game is retail-only I have no real objections either: the Phoronix Test Suite has test profiles for Doom 3, Quake 4, Counter-Strike: Source, and other pay-to-play titles. But of course for broad adoption and widespread use of the games as a benchmark, it's convenient to see the games free. If there's a demo of the game that's also better since it usually means a smaller download for making a benchmark.

If the benchmark can be automated fully and scripted, then it's very easy for me to incorporate it into the Phoronix Test Suite. Besides being my own software, Phoronix Test Suite profiles are useful beyond being automated because:

- All the commands and processes are fully standardized. There's also uniform graphs and automated hardware/software reporting.

- Numerous features like measuring standard deviation and automatically increasing the number of times to run a benchmark if a threshold is exceeded, automatic suspending of screensavers for benchmarks, logging of system data, supporting sensor monitoring when running tests, etc.

- By having a Phoronix Test Suite test profile of the game it's generally then pulled in by a wide range of organization and companies that rely upon PTS for facilitating their internal QA and performance tests. Numerous companies will then automatically begin running benchmarks with the said game/application.

- Tests can be shared and compared via the unique OpenBenchmarking.org platform.

If the workload can be automated, it's very easy to write a test profile. For a look at raw test profiles look inside (and feel free to modify) ~/.phoronix-test-suite/test-profiles/pts/. Here's some examples:

- Xonotic is a complete but a more complex test profile example that takes full advantage of all PTS features, works on not only Linux but also OS X and Windows, and automates everything from downloading of files to handling of the graphics settings.

- Team Fortress 2 for a Steam game example on Linux.

- C-Ray for a simple non-game test profile example.

- And the hundreds of other tests.

So with all of that said if there's some new game or other application you want to see benchmarked on Phoronix, here's the following recommendations:

- The most ideal path is if you -- whether you're an independent user or the creator of the software itself -- is to write your own test profile. There's many benefits in doing so and will ensure many users of the said game/software for testing purposes. Even if you don't make the test profile yourself, if you can send over a simple script that yields an example of an automated run, from there it's very easy for me to turn it into a proper Phoronix Test Suite test profile. Feel free to contact me for more information on getting involved or how to make a test profile; it's actually quite easy but admittedly there's some documentation lacking so feel free to contact me.

- If you aren't going to submit a test case yourself but are just making a request, first do your due diligence. For example, in recent weeks I've had countless requests of having Metro: Last Light benchmarks on Linux... And I've had to respond as many times -- via the forums, Twitter, etc -- that the Linux version of Metro: Last Light in Steam doesn't even have the benchmarking mode that's found on Windows. So make sure the game can at least be fully-automated and suits my requirements before trying to get me to run said test(s).

- If you really would like a test to be used and aren't willing to make it yourself, hopefully you subscribe to Phoronix Premium or make your recommendation with a PayPal contribution. I work seven days per week, about 100 hours per week, and am writing hundreds of Phoronix articles per month beyond my other job roles and PTS workloads. If you're making a request, it's higher on my TODO list if I'm motivated to try to find the time to explore such a test.

- For companies and organizations seeking custom test development and other custom engineering services, there are commercial PTS opportunities.

About The Author
Michael Larabel is the principal author of Phoronix.com and founded the web-site in 2004 with a focus on enriching the Linux hardware experience and being the largest web-site devoted to Linux hardware reviews, particularly for products relevant to Linux gamers and enthusiasts but also commonly reviewing servers/workstations and embedded Linux devices. Michael has written more than 10,000 articles covering the state of Linux hardware support, Linux performance, graphics hardware drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org automated testing software. He can be followed via and or contacted via .
Latest Linux Hardware Reviews
  1. AMD R600g/RadeonSI Performance On Linux 3.16 With Mesa 10.3-devel
  2. Intel Pentium G3258 On Linux
  3. SilverStone Precision PS10
  4. ASRock Z97 Extreme6
Latest Linux Articles
  1. KVM Benchmarks On Ubuntu 14.10
  2. X.Org Server 1.16 Officially Released With Terrific Features
  3. Ubuntu With Linux 3.16 Smashes OS X 10.9.4 On The MacBook Air
  4. Preview: Benchmarking CentOS 7.0 & Scientific Linux 7.0
Latest Linux News
  1. FreeBSD Developing A New Automounter, Other Features
  2. Guix 0.7 Can Now Install The GNU Operating System
  3. X.Org Foundation Still Works Towards Becoming An SPI Project
  4. Open-Source AMD Hawaii Support Should Now Be Working!
  5. KDE Developers Continue Working Toward Wayland Support
  6. Ubuntu 14.04.1 LTS Released
  7. Linux Developers Jump Quickly On ACPI 5.1, Helps Out ARM
  8. Pkg 1.3.0 Released To Improve Package Management On FreeBSD
  9. GOG.com Officially Starts Rolling Out Linux Games
  10. Fedora 21 Has Been Delayed By Three Weeks
Latest Forum Discussions
  1. Updated and Optimized Ubuntu Free Graphics Drivers
  2. AMD "Hawaii" Open-Source GPU Acceleration Still Not Working Right
  3. Radeon related kernel bug??
  4. how the US intellegentia operates:
  5. AMD Publishes Open-Source Linux HSA Kernel Driver
  6. Next-Gen OpenGL To Be Announced Next Month
  7. Open-Source Radeon Performance Boosted By Linux 3.16
  8. Remote gui not accessible in Phoronix Test Suite 5.2