RFC: Seamless OpenBenchmarking.org Comparisons For The Phoronix Test Suite
With the existing Phoronix Test Suite workflow, when you would be running a benchmark you wouldn't see how your performance compares to others unless you were first running a comparison against an existing result file or otherwise until after the tests have completed and you have uploaded your results to OpenBenchmarking.org, where you can then easily compare that result file against other compatible results. With the latest Phoronix Test Suite development code, it provides seamless integration with the Phoronix Test Suite to show you a real-time comparison of your current test result against automatically-selected interesting OpenBenchmarking.org results.
Built off the infrastructure for Phoronix Test Suite 7.4's progressive result reporting now has the Phoronix Test Suite querying a new OpenBenchmarking.org API to obtain automatically interesting results that are comparable and shows how your own system stacks up to those. Even if you are not saving your results, this feature will still show you these matches/comparisons.
Your results and system information are still kept locally and not automatically uploaded to OpenBenchmarking.org unless you opt so at the end of the benchmarking process, as has always been the case. With this feature, the Phoronix Test Suite only exposes to OpenBenchmarking.org the test hash being run (the unique identifier based on the test profile / test version / test parameters, for ensuring a comparable result match) and the type of system being run (whether PTS has identified the system as embedded / cloud / server / workstation / desktop -- since if you are running benchmarks on a Raspberry Pi, you probably don't want to see performance results from Amazon EC2 or a Xeon workstation...). OpenBenchmarking.org will then automatically determine the best batch of results to send to the client for display alongside your current test result.
This is only for displaying in the CLI output and isn't automatically saving these comparison results or adding it to your result file, etc. Basically for quick display purposes if wanting to see how a given benchmark's performance on your system compares to others. So with a command as simple as phoronix-test-suite benchmark encode-mp3, you can see how your own system compares to others without even needing to save the results or upload your results to the Internet. Also displayed alongside this dynamic comparison is an OpenBenchmarking.org URL if wanting to learn more about those other systems under test.
This feature is automatically enabled if running with a fast Internet connection, if you are not running in a batch/auto-mode (such as via batch-run or triggered via Phoromatic), and if you are not already running your own comparison against saved results in which case you are already likely seeing the comparison data you want.
Anyhow, that's the basics of this newest feature to OpenBenchmarking.org and to leverage the mass amounts of data already on OpenBenchmarking.org. I will still be tweaking OB's selection algorithm and such for matching to the "best" / "most interesting" results still in the days ahead, but if anyone has any comments or feedback on this feature, feel free to contact us or comment in the forums. (And if any PTS user prefers not seeing the data, they can phoronix-test-suite unload-module ob_auto_compare, as the functionality is just implemented as a new module.)
2 Comments
