How Linux Benchmarking Will Change With Iveland & OpenBenchmarking.org

Phoronix Test Suite 3.0 (codenamed "Iveland") has been under heavy development for more than a month and there is still at least three more months left of work before this major release will be christened. Today though it is time to publicly share the first details (aside from those that learned about it in the Augustiner tent at Oktoberfest) for one of the new components to be making up a critical piece of the Phoronix Test Suite 3.0 platform: OpenBenchmarking.org.

As was shared in my Phoronix Test Suite 3.0 development notes in early September, "A brand new, entirely redesigned "Phoronix Global" replacement to http://global.phoronix.com/ will be coming in tandem with Phoronix Test Suite 3.0 as part of Iveland's developments." OpenBenchmarking.org is to serve as this replacement to Phoronix Global, but with many more features to far trump its current capabilities.

OpenBenchmarking.org is the next-generation Phoronix Global platform for providing public and private storage of test result data, an integrated packaging system for remotely fetching and managing new test profiles and suites that comply with the OpenBenchmarking.org test specification, and a robust set of web-based tools for providing effective collaboration and analytical capabilities of Phoronix Test Suite results.

Essentially, OpenBenchmarking.org is the rebuild of Phoronix Global with all of the features (plus many more) that we have desired from this system since before Phoronix Test Suite 1.0 was released. Phoronix Global will continue to live around for those running a pre-Iveland Phoronix Test Suite 1.0/2.0 release, but OpenBenchmarking.org will be the central resource for those running Phoronix Test Suite 3.0 and later. The features and capabilities of the OpenBenchmarking.org software are radically different and far more advanced than that of Phoronix Global with a complete rewrite of the code-base taking place.

With OpenBenchmarking.org building in a packaging system that is seamlessly integrated into the Phoronix Test Suite client, there will no longer be any suites or test profiles actually shipping with the Phoronix Test Suite; all tests and suites will all be living in the OpenBenchmarking.org cloud. When you go to install a new test or suite, the relevant data will be automatically pulled from the OpenBenchmarking.org servers. This new method for handling of test profiles and suites provides several benefits, including the ability to immediately pull a new test profile for a given software component without needing to upgrade the entire Phoronix Test Suite package (or wait around for a new Phoronix Test Suite release), the ability to maintain the same exact test profile versions when upgrading the Phoronix Test Suite itself (for those organizations that may "lock" themselves into a particular set of test profile versions they wish to maintain for an extended period of time), and for third-parties to easily publish their own tests and suites while making it just as easy for any other individual to install these custom/third-party packages.

All of the core Phoronix Global features will remain intact with OpenBenchmarking.org, including the ability to easily benchmark your system(s) against another user's by simply knowing the result ID number, the ability to upload your test data anonymously, etc. There will also finally be the introduction of "shopping-cart-like comparisons" of test results that can all be done through the OpenBenchmarking.org web interface in a similar way to how it is handled with Phoromatic. With OpenBenchmarking.org there will also be the capabilities to better exchange result data and information between this global repository and the Phoromatic remote test management system and Phoromatic Tracker for better tracking potential performance regressions in a software stack. There will also be features to better assist those that may create a test profile simply to quickly and easily exhibit a regression -- whether it is performance-related or a unit test -- in an automated manner for any piece of software.

As part of the OpenBenchmarking.org initiative with Iveland, there have also been resources invested into a set of policies to more prominently display a user's test results within the global repository and allow them to be compared to other users, while at the same time trying to automatically detect if the result data integrity was compromised via manual editing or falsified otherwise to try to report a better score for competitive reasons.

While Phoronix.com and the development of the Phoronix Test Suite are handled by separate but linked entities, of interest to many Phoronix.com readers (and the other publications that leverage this software) will be the embeddable result graphs to be found on OpenBenchmarking.org and the improvements to be found within Iveland's pts_Graph library. There has long been a desire by some to have various statistical values and other technical metrics shown within the Phoronix Test Suite result graphs found on Phoronix.com and within the Phoronix Test Suite results viewer and other places. This will happen with OpenBenchmarking.org as these embeddable graphs will be rendered in SVG / JavaScript within supported browsers (and automatically fallback to PNG/HTML in other browsers) to allow for exposing in real-time individual result values, system logs, and other data. These far more advanced graphs will also be found within the Phoronix Test Suite results viewer on the client side, which is also receiving other new features to be talked about later on in the Iveland development cycle. Any other individuals and publications can also use these OpenBenchmarking.org embeddable graphs, which should lead to the most transparent testing processes within the industry when paired with the Phoronix Test Suite client and other components.

As expressed earlier, due to these very radical changes with Phoronix Test Suite 3.0 and the introduction of OpenBenchmarking.org, Iveland will not be introduced this quarter but rather will formally enter a production state in the first quarter of 2011. This is the first time in the Phoronix Test Suite history that a major update has not come out on a quarterly basis. Alpha and beta releases, however, should begin by late November and continue through early next year. OpenBenchmarking.org is just one of several major features to be found in Phoronix Test Suite 3.0 "Iveland", so stay tuned for more articles as we talk about the different features to further drive automated, open-source, collaborative testing on Linux, Solaris, BSD, Mac OS X, and Windows platforms.