Open-Source Performance-Per-Dollar Benchmarking

Written by Michael Larabel in Hardware on 20 August 2015 at 12:17 PM EDT. 4 Comments
HARDWARE
Recently within our forums there was a request to do performance-per-dollar benchmarks and reporting. Today I came up with a way to make this process very easy and trivial, that you can do too when carrying out your own open-source Linux tests.

As mentioned in the forums, in the past with OpenBenchmarking.org I've worked on support for automated reporting of system component pricing, PC cost estimate, etc, using Amazon's shopping/affiliate API for being able to query their pricing data in real-time. However, that wasn't always accurate for a few different reasons. As it's not been a priority feature for OpenBenchmarking.org, that's been on the back-burner. Though right now I'm in the middle of working on a new and more robust version of OpenBenchmarking.org.

This morning though I came up with a simple, stop-gap measure for being able to trivially conduct performance-per-dollar/cost benchmarks using the Phoronix Test Suite. The Phoronix Test Suite has long had a system monitoring module to monitor various system hardware/software sensors while conducting tests to show that data on a per-test basis as well as an overall view at the end of the result file. An example can be seen via this morning's GTX 950 review with that sensor data as well as performance-per-Watt figures. That data was trivially generated by just setting the MONITOR= and PERFORMANCE_PER_WATT= environment variables.


I copied that system_monitor PTS module I wrote a few years ago, gutted it, and turned it into a new Phoronix Test Suite module called perf_per_dollar. What this new module does is if the COST_PERF_PER_DOLLAR environment variable is set, it will take that as the dollar value of whatever system component (or system total cost) you're wishing to compare between several test runs. Then whenever you run a test, it will append an extra graph for each test result with a graph that shows the performance-per-dollar. Thanks to the modular design of the Phoronix Test Suite, this will work for any tests. If running multiple tests, at the end of the result file it will then also generate an overall meta/composite performance-per-dollar graph.

So the workflow would be like this for a graphics card comparison:
# Pop in a GTX 950 and fire up the Linux system, and run:
COST_PERF_PER_DOLLAR=150 phoronix-test-suite benchmark unigine-valley xonotic dirt-showdown

# Pop in a GTX TITAN X and fire up the Linux system, and run:
COST_PERF_PER_DOLLAR=999 phoronix-test-suite benchmark unigine-valley xonotic dirt-showdown

etc...
Then for each run, complementing the actual raw results will be performance-per-dollar information on each of these graphics cards. Or you could also set MONITOR=all PERFORMANCE_PER_WATT=1 too if you just want a plethora of extra data points.

Here's an example run I did with the same system but just typing in two different cost points to ensure things were working. And then another reference run to ensure multiple tests were handling things fine and that the composite value at the end would appear; but obviously running a COST_PERF_PER_DOLLAR comparison on a single data point isn't too useful. After those initial tests, I then added support for reporting a footnote on each graph of the actual reported cost, example here.

This new module is now in Phoronix-Test-Suite on GitHub and will be found in the Phoronix Test Suite 6.0 stable release later the year. This is the actual commit if you want to look at the code and to see what's possible with the Phoronix Test Suite for extending it in about 80 lines of code... Any feedback or suggestions on further extending this performance-per-dollar support is welcome.

This simple but easy-to-use feature will hopefully be useful to you too and just another of reasons you should make a PTS test profile for your own software or other software you care about for performance/testing reasons... Adding complementary performance-per-dollar benchmarks will now be more common in future Linux hardware reviews now that it just takes a few seconds adding this COST_PERF_PER_DOLLAR= environment variable prior to each benchmark run on a new comparison.

With that said, in the next few days you can expect to see some Phoronix articles utilizing this feature in looking at graphics cards for delivering the best value and "bang for your buck" on Linux/SteamOS. Thanks to Premium subscribers and tippers this week for the extra motivation to get this open-source module in order for our GPL licensed benchmarking software.
Related News
About The Author
Michael Larabel

Michael Larabel is the principal author of Phoronix.com and founded the site in 2004 with a focus on enriching the Linux hardware experience. Michael has written more than 20,000 articles covering the state of Linux hardware support, Linux performance, graphics drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org automated benchmarking software. He can be followed via Twitter, LinkedIn, or contacted via MichaelLarabel.com.

Popular News This Week