Phoronix Test Suite Released

Back in early February we announced that we were in the process of formalizing and releasing our internal test tools as a platform for facilitating easy to use, accurate, and reproducible Linux benchmarks based upon the testing work that we have been doing at Phoronix for the past four years. The goals with this are really to make it easier for Linux end-users to run reliable (both qualitative and quantitative) benchmarks for their own personal use, push more open-source projects to making their software more testable, and pushing hardware and software vendors for greater Linux testing based upon a standardized set of tests. Today we are very pleased to announce the first public release of the Phoronix Test Suite software, which is licensed under the GNU GPLv3, and the creation of a public database for freely sharing your test results and other information in a collaborative manner.

The Phoronix Test Suite is not just a set of benchmarking scripts that we have used internally for conducting our hardware benchmarking that we are now opening up, but it's quite the opposite. In fact, the Phoronix Test Suite code contains zero benchmark-specific conditions. The Phoronix Test Suite is a testing and benchmarking platform for carrying out Linux benchmarks and layered on top of that are XML-based benchmarking profiles that contain the work specific to each test. With this initial release, we are including a simple set of benchmarks that provide some immediate testing, but these tests can be easily extended. This is a culmination of our Linux benchmarking procedures that have evolved over the past four years at Phoronix with input from tier-one hardware vendors.

Among its many benefits, the Phoronix Test Suite is able to automatically recognize important system attributes such as the number of CPU cores present and adjust accordingly in such tests where it can make a difference -- such as the number of jobs during GCC compilation tests and benchmarks that have SMP-optimized binaries. Further differentiating the Phoronix Test Suite from just a set of benchmarking scripts is that once all tests have been completed, the results are cleanly displayed within the Phoronix Test Suite Results Viewer showing all relevant information and results from each test in both text and graph form. In addition, the system's important hardware and software information is automatically parsed and stored with the results. The entire process from start to finish is handled in a heavily automated and completely repeatable manner -- this includes the installation process too. If the Phoronix Test Suite detects a benchmark that isn't installed on the target system (and the profile maintainer has written a PTS install script), the Phoronix Test Suite will install the benchmark and any needed dependencies in a distribution-neutral fashion.

When running the Phoronix Test Suite, a test can be ran individually or as part of a suite. A suite could be a collection of all gaming tests for instance or a select set of tests known to generally stress systems well. When running a test individually, any options exported to the XML profile are prompted (such as the resolution to run at for graphical benchmarks) followed by executing the tests. Shipping with the Phoronix Test Suite 0.1 release are 17 test profiles and 7 test suites. However, with this release the status of each profile and suite varies. While this is the first public release of the Phoronix Test Suite and it's working well in our tests, we aren't yet deploying the Phoronix Test Suite on our production benchmarking systems for Phoronix articles.

Just how easy and quick is it? The entire process from start to finish can be completed in just a matter of minutes (assuming a high-speed Internet connection). Below are some of the most common commands.

Run All Benchmarks

./phoronix-test-suite benchmark universe

Running A Global Comparison

./phoronix-test-suite benchmark <global ID>

Upload Results

./phoronix-test-suite upload <saved name>

Merge Two Test Results

./phoronix-test-suite merge-results <saved name #1> <saved name #2> <save to>

The Phoronix Test Suite is that easy to work and to produce meaningful results. On the following pages are more detailed descriptions of the Phoronix Test Suite and its capabilities, PTS Global (our public benchmarking database), the GUI and results viewer, and the road-map for this project.