1. Computers
  2. Display Drivers
  3. Graphics Cards
  4. Memory
  5. Motherboards
  6. Processors
  7. Software
  8. Storage
  9. Operating Systems


Facebook RSS Twitter Twitter Google Plus


Phoronix Test Suite

OpenBenchmarking.org

Finally, Team Fortress 2 Benchmarks For Phoronix!

Gaming

Published on 18 July 2013 12:39 AM EDT
Written by Michael Larabel in Gaming
78 Comments

There's finally a Phoronix Test Suite test profile for being able to benchmark Valve's Team Fortress 2 on Steam in a (semi-) automated manner on Linux.

One of the leading requests for months has been Team Fortress 2 (and other Source Engine) benchmarks on Phoronix. While Phoronix was the first site to detail Valve's Linux plans, benchmarks of their games have been slow to come.

The principal reason for this delay (aside from the fact of the Steam client leaving beta and becoming widely accessible) is that launching the specified game directly in the past has hit some snags where the process wouldn't completely end upon completion (after using timedemoquit). The game process would basically hang even after the timed demo was complete and the window closed. Believing firmly about automated testing and reproducibility along with my Phoronix Test Suite / Phoromatic / OpenBenchmarking.org projects, this has been a problem.

After communicating with Valve and trying out the latest TF2 version on Steam, the process exiting issue has been resolved. After working around another new library path issue, the Team Fortress 2 test profile is up and running!

Finally, Team Fortress 2 Benchmarks For Phoronix!

Other gripes I have and why the Steam-based titles are seldom benchmarked and covered on Phoronix include:

- The Steam process is hard to fully automate (100%), with different per-distribution/OS caveats and needing an account for installing the games. (This is still an issue with the current Team Fortress 2 test profile and it assumes you already have Steam installed on the system.)

- As can be seen by other common benchmarks like Unigine and Lightsmark and the id Software games, I don't really care if a given workload is open-source or not. Open-source is obviously preferred, but quality benchmarks carry more weight. Free is really nice for ensuring the community can reproduce the tests and compare the published results against their own systems. Having to deal with account registration and other hurdles decreases the likelihood others will want to reproduce and/or use the given test(s).

- The game payloads aren't freely redistributable and they're huge in size. As an example, Team Fortress 2 takes up about 13GB on-disk even though we just care about running a small time demo. For test systems that are formatted frequently (can be multiple times per week or even multiple times per day when swapping out lots of hardware/software components), this is wasted bandwidth unless manually backing up the files. This can also be a problem for enterprise test farms that are "behind the firewall" and may not have public Internet access. For the freely distributable files and what's provided by most of the test profiles, the Phoronix Test Suite and its related components can take care of file/download management, locally caching the files for other Phoronix Test Suite systems by using an intranet server or USB storage device, etc. It makes the entire install/setup process very streamlined rather than needing to download gigabytes of data through Steam each time.

- When installing Team Fortress 2 or other games through Steam, they take care of auto-updating, which is nice for gamers but not for benchmarking. For the packaged benchmarks, we don't have to worry about game updates and whether a game engine is going to be revised in a given update or other game changes that could potentially affect the performance of the game compared to the existing results. There appears to basically be no way to ensure that the same exact game revision is used. (Heck, even just in the time waiting for the original TF2 issue to be resolved in the past few months, the LD_LIBRARY_PATH environment variable requirements have changes for being able to run the game.)

Those are just the main items that immediately come to mind... They're being expressed here for those curious or for other software developers looking to make their applications/games nicer for automated testing. These just aren't my personal pet peeves but are requirements as well that are mandated by other Phoronix Test Suite commercial customers and enterprises wishing to ensure 100% automation and reproducibility of their performance testing.

Finally, Team Fortress 2 Benchmarks For Phoronix!


Anyhow, the simple Team Fortress 2 test profile can be found on OpenBenchmarking.org. The test profile can be seen on this page and here's the test profile contents.

With making the assumption Steam is pre-installed along with Team Fortress 2, it basically comes down to downloading a standardized time-demo, integrating with Phoronix Test Suite various user options (for now, namely the resolution), and then proceeds to run the Team Fortress 2 benchmark in a standardized way. Normally the Phoronix Test Suite installs all of its test assets within ~/.phoronix-test-suite/installed-tests/ for ensuring checksums match, there's no extra file patches, etc, but due to each Valve game occupying several gigabytes (plus the other factors), it's reverting to just using the system's original game binaries. This test profile right now also doesn't handle OS X or Windows Steam, but support could be trivially added as the Phoronix Test Suite is compatible with all major operating systems.

The results of the Team Fortress 2 are then sent back into the Phoronix Test Suite. All of the usual Phoronix Test Suite features are available to this test profile -- and all others -- like being able to concurrently measure various hardware/software system sensors, formatted graphics, system/hardware parsing, the module framework with its features like automated screensaver disabling during testing, capturing/bundling of system logs with test results, etc.

With Steam and TF2 installed, it should just be a matter of running phoronix-test-suite benchmark tf2 for tapping into this goodness. Coming up soon will be some Team Fortress 2 benchmarks, but due to the aforementioned criteria, don't expect to see TF2 coverage in every Phoronix Linux driver/graphics article.

About The Author
Michael Larabel is the principal author of Phoronix.com and founded the web-site in 2004 with a focus on enriching the Linux hardware experience and being the largest web-site devoted to Linux hardware reviews, particularly for products relevant to Linux gamers and enthusiasts but also commonly reviewing servers/workstations and embedded Linux devices. Michael has written more than 10,000 articles covering the state of Linux hardware support, Linux performance, graphics hardware drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org automated testing software. He can be followed via and or contacted via .
Latest Linux Hardware Reviews
  1. AMD's New Athlon/Semprons Give Old Phenom CPUs A Big Run For The Money
  2. 13-Way Low-End GPU Comparison With AMD's AM1 Athlon
  3. ASUS AM1I-A: A Mini-ITX Board For Socketed Kabini APUs
  4. Mini-Box M350: A Simple, Affordable Mini-ITX Case
Latest Linux Articles
  1. How Much Video RAM Is Needed For Catalyst R3 Graphics?
  2. Ubuntu 12.04 LTS vs. 14.04 LTS Cloud Benchmarks
  3. Ubuntu 12.04.4 vs. 13.10 vs. 14.04 LTS Desktop Benchmarks
  4. AMD OpenCL Performance With AM1 Kabini APUs
Latest Linux News
  1. OpenELEC 4.0 Beta 6 Works On 4K Graphics, RPi ALSA
  2. Linux 3.15 Lands Some DRM Graphics Driver Fixes
  3. AMD Is Disabling DPM Support For RV770 GPUs
  4. ReactOS Working On A Community Windows OS
  5. Borderlands Is Being Considered For Linux
  6. Mesa 10.0 & 10.1 Stable Get Updated
  7. Getting Hit By The Variable Performance Of The Public Cloud
  8. Git 2.0 Test Releases Begin With Many Changes
  9. Wine 1.7.17 Works On Its Task Scheduler, C Run-Time
  10. The Improv ARM Board Still Isn't Shipping; Riding A Dead Horse?
  11. Debian To Maintain 6.0 Squeeze As An LTS Release
  12. Wasteland 2 Is Finally Released For Linux Gamers
Latest Forum Discussions
  1. The GNOME Foundation Is Running Short On Money
  2. Updated and Optimized Ubuntu Free Graphics Drivers
  3. Catalyst 14.3 Beta
  4. Suggestions about how to make a Radeon HD 7790 work decently?
  5. Radeon 8000M problematic on Linux?
  6. Linux Kernel Developers Fed Up With Ridiculous Bugs In Systemd
  7. After Jack Keane, RuseSoft will briing Ankh 3 to Linux through Desura
  8. Suspected PHP Proxy Issue