1. Computers
  2. Display Drivers
  3. Graphics Cards
  4. Memory
  5. Motherboards
  6. Processors
  7. Software
  8. Storage
  9. Operating Systems

Facebook RSS Twitter Twitter Google Plus

Phoronix Test Suite

OpenBenchmarking Benchmarking Platform
Phoromatic Test Orchestration

Finally, Team Fortress 2 Benchmarks For Phoronix!


Published on 18 July 2013 12:39 AM EDT
Written by Michael Larabel in Gaming

There's finally a Phoronix Test Suite test profile for being able to benchmark Valve's Team Fortress 2 on Steam in a (semi-) automated manner on Linux.

One of the leading requests for months has been Team Fortress 2 (and other Source Engine) benchmarks on Phoronix. While Phoronix was the first site to detail Valve's Linux plans, benchmarks of their games have been slow to come.

The principal reason for this delay (aside from the fact of the Steam client leaving beta and becoming widely accessible) is that launching the specified game directly in the past has hit some snags where the process wouldn't completely end upon completion (after using timedemoquit). The game process would basically hang even after the timed demo was complete and the window closed. Believing firmly about automated testing and reproducibility along with my Phoronix Test Suite / Phoromatic / OpenBenchmarking.org projects, this has been a problem.

After communicating with Valve and trying out the latest TF2 version on Steam, the process exiting issue has been resolved. After working around another new library path issue, the Team Fortress 2 test profile is up and running!

Finally, Team Fortress 2 Benchmarks For Phoronix!

Other gripes I have and why the Steam-based titles are seldom benchmarked and covered on Phoronix include:

- The Steam process is hard to fully automate (100%), with different per-distribution/OS caveats and needing an account for installing the games. (This is still an issue with the current Team Fortress 2 test profile and it assumes you already have Steam installed on the system.)

- As can be seen by other common benchmarks like Unigine and Lightsmark and the id Software games, I don't really care if a given workload is open-source or not. Open-source is obviously preferred, but quality benchmarks carry more weight. Free is really nice for ensuring the community can reproduce the tests and compare the published results against their own systems. Having to deal with account registration and other hurdles decreases the likelihood others will want to reproduce and/or use the given test(s).

- The game payloads aren't freely redistributable and they're huge in size. As an example, Team Fortress 2 takes up about 13GB on-disk even though we just care about running a small time demo. For test systems that are formatted frequently (can be multiple times per week or even multiple times per day when swapping out lots of hardware/software components), this is wasted bandwidth unless manually backing up the files. This can also be a problem for enterprise test farms that are "behind the firewall" and may not have public Internet access. For the freely distributable files and what's provided by most of the test profiles, the Phoronix Test Suite and its related components can take care of file/download management, locally caching the files for other Phoronix Test Suite systems by using an intranet server or USB storage device, etc. It makes the entire install/setup process very streamlined rather than needing to download gigabytes of data through Steam each time.

- When installing Team Fortress 2 or other games through Steam, they take care of auto-updating, which is nice for gamers but not for benchmarking. For the packaged benchmarks, we don't have to worry about game updates and whether a game engine is going to be revised in a given update or other game changes that could potentially affect the performance of the game compared to the existing results. There appears to basically be no way to ensure that the same exact game revision is used. (Heck, even just in the time waiting for the original TF2 issue to be resolved in the past few months, the LD_LIBRARY_PATH environment variable requirements have changes for being able to run the game.)

Those are just the main items that immediately come to mind... They're being expressed here for those curious or for other software developers looking to make their applications/games nicer for automated testing. These just aren't my personal pet peeves but are requirements as well that are mandated by other Phoronix Test Suite commercial customers and enterprises wishing to ensure 100% automation and reproducibility of their performance testing.

Finally, Team Fortress 2 Benchmarks For Phoronix!

Anyhow, the simple Team Fortress 2 test profile can be found on OpenBenchmarking.org. The test profile can be seen on this page and here's the test profile contents.

With making the assumption Steam is pre-installed along with Team Fortress 2, it basically comes down to downloading a standardized time-demo, integrating with Phoronix Test Suite various user options (for now, namely the resolution), and then proceeds to run the Team Fortress 2 benchmark in a standardized way. Normally the Phoronix Test Suite installs all of its test assets within ~/.phoronix-test-suite/installed-tests/ for ensuring checksums match, there's no extra file patches, etc, but due to each Valve game occupying several gigabytes (plus the other factors), it's reverting to just using the system's original game binaries. This test profile right now also doesn't handle OS X or Windows Steam, but support could be trivially added as the Phoronix Test Suite is compatible with all major operating systems.

The results of the Team Fortress 2 are then sent back into the Phoronix Test Suite. All of the usual Phoronix Test Suite features are available to this test profile -- and all others -- like being able to concurrently measure various hardware/software system sensors, formatted graphics, system/hardware parsing, the module framework with its features like automated screensaver disabling during testing, capturing/bundling of system logs with test results, etc.

With Steam and TF2 installed, it should just be a matter of running phoronix-test-suite benchmark tf2 for tapping into this goodness. Coming up soon will be some Team Fortress 2 benchmarks, but due to the aforementioned criteria, don't expect to see TF2 coverage in every Phoronix Linux driver/graphics article.

About The Author
Michael Larabel is the principal author of Phoronix.com and founded the web-site in 2004 with a focus on enriching the Linux hardware experience and being the largest web-site devoted to Linux hardware reviews, particularly for products relevant to Linux gamers and enthusiasts but also commonly reviewing servers/workstations and embedded Linux devices. Michael has written more than 10,000 articles covering the state of Linux hardware support, Linux performance, graphics hardware drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org automated testing software. He can be followed via and or contacted via .
Latest Linux News
  1. Global Shortcuts In KDE Plasma Under Wayland
  2. LLVMpipe FP64 Support Knocks Off Some GL4 Extensions
  3. Dell Gets An Airplane Mode Switch Driver In Linux 4.2
  4. I Gave Up Waiting On The Water-Cooled Radeon R9 Fury X
  5. NVIDIA Tegra X1 Chromebooks Appear Closer, Support Added To Coreboot
  6. Pinos Is For Linux Video What PulseAudio Is For Audio
  7. Crossing 200,000 Benchmark Results Posted On LinuxBenchmarking.com
  8. New Mesa Vec4 Backend For Intel, Supports Their NIR Goals
  9. "PulseVideo" Coming To Complement PulseAudio?
  10. Premium Users Now Can Experience Our New Site
Latest Articles & Reviews
  1. How KDE VDG Is Trying To Make Open-Source Software Beautiful
  2. Attempting To Try Out BCache On The Linux 4.1 Kernel
  3. CompuLab's Fitlet Is A Very Tiny, Fanless, Linux PC With AMD A10 Micro
  4. AMD A10-7870K Godavari: RadeonSI Gallium3D vs. Catalyst Linux Drivers
Most Viewed News This Week
  1. Kubuntu 15.10 Could Be The End Of The Road
  2. NVIDIA Starts Supplying Open-Source Hardware Reference Headers
  3. KDBUS Won't Be Pushed Until The Linux 4.3 Kernel
  4. The Staging Pull For Linux 4.2: "Big, Really Big"
  5. The State & Complications Of Porting The Unity Editor To Linux
  6. SteamOS "Brewmaster" Is Valve's New Debian 8.1 Based Version
  7. Jonathan Riddell Steps Down From The Kubuntu Council
  8. ARM Posts Pictures Of AMD's New Development Board