1. Computers
  2. Display Drivers
  3. Graphics Cards
  4. Memory
  5. Motherboards
  6. Processors
  7. Software
  8. Storage
  9. Operating Systems

Facebook RSS Twitter Twitter Google Plus

Phoronix Test Suite

OpenBenchmarking Benchmarking Platform
Phoromatic Test Orchestration

Finally, Team Fortress 2 Benchmarks For Phoronix!


Published on 18 July 2013 12:39 AM EDT
Written by Michael Larabel in Gaming

There's finally a Phoronix Test Suite test profile for being able to benchmark Valve's Team Fortress 2 on Steam in a (semi-) automated manner on Linux.

One of the leading requests for months has been Team Fortress 2 (and other Source Engine) benchmarks on Phoronix. While Phoronix was the first site to detail Valve's Linux plans, benchmarks of their games have been slow to come.

The principal reason for this delay (aside from the fact of the Steam client leaving beta and becoming widely accessible) is that launching the specified game directly in the past has hit some snags where the process wouldn't completely end upon completion (after using timedemoquit). The game process would basically hang even after the timed demo was complete and the window closed. Believing firmly about automated testing and reproducibility along with my Phoronix Test Suite / Phoromatic / OpenBenchmarking.org projects, this has been a problem.

After communicating with Valve and trying out the latest TF2 version on Steam, the process exiting issue has been resolved. After working around another new library path issue, the Team Fortress 2 test profile is up and running!

Finally, Team Fortress 2 Benchmarks For Phoronix!

Other gripes I have and why the Steam-based titles are seldom benchmarked and covered on Phoronix include:

- The Steam process is hard to fully automate (100%), with different per-distribution/OS caveats and needing an account for installing the games. (This is still an issue with the current Team Fortress 2 test profile and it assumes you already have Steam installed on the system.)

- As can be seen by other common benchmarks like Unigine and Lightsmark and the id Software games, I don't really care if a given workload is open-source or not. Open-source is obviously preferred, but quality benchmarks carry more weight. Free is really nice for ensuring the community can reproduce the tests and compare the published results against their own systems. Having to deal with account registration and other hurdles decreases the likelihood others will want to reproduce and/or use the given test(s).

- The game payloads aren't freely redistributable and they're huge in size. As an example, Team Fortress 2 takes up about 13GB on-disk even though we just care about running a small time demo. For test systems that are formatted frequently (can be multiple times per week or even multiple times per day when swapping out lots of hardware/software components), this is wasted bandwidth unless manually backing up the files. This can also be a problem for enterprise test farms that are "behind the firewall" and may not have public Internet access. For the freely distributable files and what's provided by most of the test profiles, the Phoronix Test Suite and its related components can take care of file/download management, locally caching the files for other Phoronix Test Suite systems by using an intranet server or USB storage device, etc. It makes the entire install/setup process very streamlined rather than needing to download gigabytes of data through Steam each time.

- When installing Team Fortress 2 or other games through Steam, they take care of auto-updating, which is nice for gamers but not for benchmarking. For the packaged benchmarks, we don't have to worry about game updates and whether a game engine is going to be revised in a given update or other game changes that could potentially affect the performance of the game compared to the existing results. There appears to basically be no way to ensure that the same exact game revision is used. (Heck, even just in the time waiting for the original TF2 issue to be resolved in the past few months, the LD_LIBRARY_PATH environment variable requirements have changes for being able to run the game.)

Those are just the main items that immediately come to mind... They're being expressed here for those curious or for other software developers looking to make their applications/games nicer for automated testing. These just aren't my personal pet peeves but are requirements as well that are mandated by other Phoronix Test Suite commercial customers and enterprises wishing to ensure 100% automation and reproducibility of their performance testing.

Finally, Team Fortress 2 Benchmarks For Phoronix!

Anyhow, the simple Team Fortress 2 test profile can be found on OpenBenchmarking.org. The test profile can be seen on this page and here's the test profile contents.

With making the assumption Steam is pre-installed along with Team Fortress 2, it basically comes down to downloading a standardized time-demo, integrating with Phoronix Test Suite various user options (for now, namely the resolution), and then proceeds to run the Team Fortress 2 benchmark in a standardized way. Normally the Phoronix Test Suite installs all of its test assets within ~/.phoronix-test-suite/installed-tests/ for ensuring checksums match, there's no extra file patches, etc, but due to each Valve game occupying several gigabytes (plus the other factors), it's reverting to just using the system's original game binaries. This test profile right now also doesn't handle OS X or Windows Steam, but support could be trivially added as the Phoronix Test Suite is compatible with all major operating systems.

The results of the Team Fortress 2 are then sent back into the Phoronix Test Suite. All of the usual Phoronix Test Suite features are available to this test profile -- and all others -- like being able to concurrently measure various hardware/software system sensors, formatted graphics, system/hardware parsing, the module framework with its features like automated screensaver disabling during testing, capturing/bundling of system logs with test results, etc.

With Steam and TF2 installed, it should just be a matter of running phoronix-test-suite benchmark tf2 for tapping into this goodness. Coming up soon will be some Team Fortress 2 benchmarks, but due to the aforementioned criteria, don't expect to see TF2 coverage in every Phoronix Linux driver/graphics article.

About The Author
Michael Larabel is the principal author of Phoronix.com and founded the web-site in 2004 with a focus on enriching the Linux hardware experience and being the largest web-site devoted to Linux hardware reviews, particularly for products relevant to Linux gamers and enthusiasts but also commonly reviewing servers/workstations and embedded Linux devices. Michael has written more than 10,000 articles covering the state of Linux hardware support, Linux performance, graphics hardware drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org automated testing software. He can be followed via and or contacted via .
Latest Linux News
  1. Fedora 22 Risks Being Delayed Beyond Next Week
  2. Systemd 220 Has Finally Been Released
  3. LibreOffice 5.0 Beta 1 Released
  4. Allwinner Publishes New CedarX Open-Source Code
  5. ACPI 6 Non-Volatile Memory Device Support / NFIT / LIBND For Linux
  6. Fedora 22 Marches Closer To Release
  7. The Linux 4.0 EXT4 RAID Corruption Bug Has Been Uncovered
  8. Ubuntu 15.10 Release Schedule Firmed Up
  9. Will Ubuntu Linux Hit 200 Million Users This Year?
  10. Microsoft Open-Sources The Windows Communication Foundation
  11. Intel Skylake Adds ASTC Texture Compression, Open-Source Support Coming
  12. Qt Turns 20 Years Old, KDE Celebrates
Latest Articles & Reviews
  1. Btrfs RAID 0/1 Benchmarks On The Linux 4.1 Kernel
  2. The State Of Various Firefox Features
  3. Intel Iris Graphics Performance With Mesa 10.6
  4. Fedora Workstation 22 Is Looking Great, Running Fantastic
Most Viewed News This Week
  1. The Linux 4.0 Kernel Currently Has An EXT4 Corruption Issue
  2. Rust 1.0 Language Officially Released
  3. AMDGPU Open-Source Driver Code Continues Maturing
  4. Oculus Rift Suspends Linux Development To Focus On Windows
  5. Wine 1.7.43 Works On Desktop Shell Window Support
  6. Spec Ops: The Line Is The Latest Linux Shooter
  7. RadeonSI Gallium3D Driver To Be Enabled For Android
  8. KDE Applications 15.08 Planned For Release On 19 August