Finally, Team Fortress 2 Benchmarks For Phoronix!
One of the leading requests for months has been Team Fortress 2 (and other Source Engine) benchmarks on Phoronix. While Phoronix was the first site to detail Valve's Linux plans, benchmarks of their games have been slow to come.
The principal reason for this delay (aside from the fact of the Steam client leaving beta and becoming widely accessible) is that launching the specified game directly in the past has hit some snags where the process wouldn't completely end upon completion (after using timedemoquit). The game process would basically hang even after the timed demo was complete and the window closed. Believing firmly about automated testing and reproducibility along with my Phoronix Test Suite / Phoromatic / OpenBenchmarking.org projects, this has been a problem.
After communicating with Valve and trying out the latest TF2 version on Steam, the process exiting issue has been resolved. After working around another new library path issue, the Team Fortress 2 test profile is up and running!
- The Steam process is hard to fully automate (100%), with different per-distribution/OS caveats and needing an account for installing the games. (This is still an issue with the current Team Fortress 2 test profile and it assumes you already have Steam installed on the system.)
- As can be seen by other common benchmarks like Unigine and Lightsmark and the id Software games, I don't really care if a given workload is open-source or not. Open-source is obviously preferred, but quality benchmarks carry more weight. Free is really nice for ensuring the community can reproduce the tests and compare the published results against their own systems. Having to deal with account registration and other hurdles decreases the likelihood others will want to reproduce and/or use the given test(s).
- The game payloads aren't freely redistributable and they're huge in size. As an example, Team Fortress 2 takes up about 13GB on-disk even though we just care about running a small time demo. For test systems that are formatted frequently (can be multiple times per week or even multiple times per day when swapping out lots of hardware/software components), this is wasted bandwidth unless manually backing up the files. This can also be a problem for enterprise test farms that are "behind the firewall" and may not have public Internet access. For the freely distributable files and what's provided by most of the test profiles, the Phoronix Test Suite and its related components can take care of file/download management, locally caching the files for other Phoronix Test Suite systems by using an intranet server or USB storage device, etc. It makes the entire install/setup process very streamlined rather than needing to download gigabytes of data through Steam each time.
- When installing Team Fortress 2 or other games through Steam, they take care of auto-updating, which is nice for gamers but not for benchmarking. For the packaged benchmarks, we don't have to worry about game updates and whether a game engine is going to be revised in a given update or other game changes that could potentially affect the performance of the game compared to the existing results. There appears to basically be no way to ensure that the same exact game revision is used. (Heck, even just in the time waiting for the original TF2 issue to be resolved in the past few months, the LD_LIBRARY_PATH environment variable requirements have changes for being able to run the game.)
Those are just the main items that immediately come to mind... They're being expressed here for those curious or for other software developers looking to make their applications/games nicer for automated testing. These just aren't my personal pet peeves but are requirements as well that are mandated by other Phoronix Test Suite commercial customers and enterprises wishing to ensure 100% automation and reproducibility of their performance testing.
Anyhow, the simple Team Fortress 2 test profile can be found on OpenBenchmarking.org. The test profile can be seen on this page and here's the test profile contents.
With making the assumption Steam is pre-installed along with Team Fortress 2, it basically comes down to downloading a standardized time-demo, integrating with Phoronix Test Suite various user options (for now, namely the resolution), and then proceeds to run the Team Fortress 2 benchmark in a standardized way. Normally the Phoronix Test Suite installs all of its test assets within ~/.phoronix-test-suite/installed-tests/ for ensuring checksums match, there's no extra file patches, etc, but due to each Valve game occupying several gigabytes (plus the other factors), it's reverting to just using the system's original game binaries. This test profile right now also doesn't handle OS X or Windows Steam, but support could be trivially added as the Phoronix Test Suite is compatible with all major operating systems.
The results of the Team Fortress 2 are then sent back into the Phoronix Test Suite. All of the usual Phoronix Test Suite features are available to this test profile -- and all others -- like being able to concurrently measure various hardware/software system sensors, formatted graphics, system/hardware parsing, the module framework with its features like automated screensaver disabling during testing, capturing/bundling of system logs with test results, etc.
With Steam and TF2 installed, it should just be a matter of running phoronix-test-suite benchmark tf2 for tapping into this goodness. Coming up soon will be some Team Fortress 2 benchmarks, but due to the aforementioned criteria, don't expect to see TF2 coverage in every Phoronix Linux driver/graphics article.
78 Comments
