Going Beyond Just Measuring Frame Rates

Yesterday marked the release of Phoronix Test Suite 2.2 and it was the best version yet with the addition of many new exciting and useful features. While this release was gratifying, there are much greater plans for the Phoronix Test Suite going into the next decade. It has already been shared that Windows support is coming, but there are other huge features coming too as soon as Q1'2010. Up to this point, most of the tests and the design of pts-core (the Phoronix Test Suite engine) have been focused on quantitative benchmarks with many of the tests spitting out a frame-rate, time, or some other measurement. However, now being supported in the Phoronix Test Suite is the ability to produce abstract results, such as screenshots used for image quality comparisons. The Phoronix Test Suite can now track the image quality of various test profiles (such as OpenGL games) across hardware configurations, drivers, and more. All of this is still leveraged upon the existing Phoronix Test Suite framework and our design philosophies so that even image quality comparisons can be carried out autonomously, the ability to compare many results side-by-side, support for carrying out these tests remotely via Phoromatic, and the ability to share your abstract results with others via Phoronix Global. Now not only can you be sure you are satisfied with the quantitative frame-rate of the hardware you have -- or are about to purchase -- but you have a plethora of options for looking at the qualitative performance too.

Right now most of the "abstract results" work that is already completed for Phoronix Test Suite 2.4 is for image quality comparisons, but other result types will continue to be added. For end-users, this allows them to not only use our software to find out the speed of a graphics card via an assortment of games and other 2D/3D tests found in the Phoronix Test Suite (there's 41 graphics test profiles right now!), but to also find out how the image quality compares between drivers and graphics cards. We intend to begin providing these image quality results within graphics card reviews and driver comparisons at Phoronix.com and hope the other publications using the Phoronix Test Suite will do the same as well. In an open letter to tech review sites we called out on fellow web-sites to begin testing hardware under Linux to complement their Windows testing considering the ease-of-use of the Phoronix Test Suite or how even easy it can get when using our Linux distribution. With image quality comparisons this remains just as easy to carry out. Image quality comparisons can also be done through Phoronix Global for an even larger comparison base from users around the world (though automated image uploads for Phoronix Global is currently disabled due to bandwidth concerns).

To developers and quality assurance engineers, this new support is even of greater benefit. Tests that produce image quality comparisons are carried out with the Phoronix Test Suite in the very same way that all other test profiles are processed. What this means is that any existing Phoronix Test Suite commands, PTS modules (thermal monitoring, etc), and other Phoronix Test Suite components will all work with these new test profiles. Image quality tests can be scheduled remotely through our remote test management software and all of that good stuff. What this also means is that these test profiles can also leverage the immense power offered to traditional test profiles producing quantitative results.

Below is an example rendering that was produced by the Phoronix Test Suite -- with our latest Phoronix Test Suite 2.4 "Lenvik" code as of today -- in its usual autonomous, clean, reproducible, and easy-to-use manner. This is an example from the iqc-nexuiz test profile, which runs Image Quality Comparison (IQC) tests atop the popular Nexuiz game. When running this test profile the user is prompted with what scene(s) to capture and the Phoronix Test Suite is off doing its magic. This test profile is one of the simplest yet most advanced test profiles found in the Phoronix Test Suite through its use of features like Cascading Test Profiles, cache shares, and other pts-core technologies.

This image (which was a simple save-as from the PTS Results Viewer and then compressed in The GIMP to conserve bandwidth) was produced by simply running phoronix-test-suite benchmark iqc-nexuiz and then running phoronix-test-suite benchmark iqc-nexuiz after installing a knowingly faulty driver configuration, as an example to show off the difference between two images. Easy as that. Those writing test profiles also have a set of options to now exploit with the Phoronix Test Suite for capturing screenshots with per-frame accuracy and storage without any compression, the automatic cropping of images based upon set data, varying image sizes, and other options. Those constructing test suites can also house quantitative and qualitative tests all within the same suite -- there is no limits being placed on qualitative tests.