Last month Phoromatic went into public beta, which is our remote test management software for the Phoronix Test Suite that allows a wealth of possibilities including the ability to easily build a benchmarking test farm. At the start of this month, we in fact announced that the Phoronix Kernel Test Farm went live and it would be benchmarking the latest mainline Linux kernel on a daily basis. This was followed by the addition of a system in our test farm to benchmark the latest Fedora Rawhide packages on a daily basis. We had not intended to begin pushing out these results publicly through a new web-site until next year, but we have already collected some interesting metrics that are documenting active regressions within the Linux 2.6.33 kernel. As a result, this morning we are rolling out Phoromatic Tracker, the public interface to our test farm.
Phoromatic is all about making it efficient and effective to manage the Phoronix Test Suite -- from the execution of the tests to the analyzing of the results -- all from a central web-interface that can scale to an unlimited number of testing machines whether they are running Linux, FreeBSD, NetBSD, OpenBSD, OpenSolaris, Mac OS X, Palm webOS, or soon Windows 7. These testing machines can be running internally within the QA department of an organization or scattered around the world. With the current implementation of Phoromatic, tests can be scheduled to run across any selected machines at a given time and then results are then immediately uploaded to the login-protected web interface. We have seen promising adoption of Phoromatic thus far, but it is still in beta and new features are continuing to be brewed.
The Phoromatic Tracker is the first extension of Phoromatic and it provides greater analytical and performance tracking capabilities of test results and, most notably, it provides an option to open up these results to the public. The kernel tracker is publicly available as of this morning and at least one more tracker will go live before month's end.
These trackers are following our Phoromatic results in real-time from our test farm and -- like all our other test components -- is all carried out autonomously and is repeatable and can be quickly and easily deployed. For the kernel tracker you can visit it nightly and see the very latest results. There are around 50 tests running with this test schedule. From the top of the page you can select a range from which to display results, if you wish to just glance at the results for the past few days or wish to look at how the performance has evolved over the course of weeks or months (when there is sufficient data built up). Right now, we just have one system running in the kernel tracker tree, but multiple systems are certainly supported and when running from the Phoromatic Tracker interface you can be viewing multiple system results simultaneously or individually.
Beyond just displaying customized results cleanly and effectively, we are continuing to build up analytical features and other advanced capabilities within the Phoronix Test Suite and Phoromatic. With the Phoromatic Tracker, one of its features is an automated regression detector. Rather than scrolling through ~50 graphs each day looking for changes, near the top of the page is a table that is automatically built in real-time and using pts-core features it attempts to spot all notable performance regressions whether they negatively or positively affect the system's results. The regression threshold can also be adjusted if you wish to just find performance regressions that exceed a defined threshold (as a percent, currently). Furthermore, this integrated regression detector also attempts to automatically filter out reporting any performance regressions that are corrected in later test runs, so you are looking at only performance regressions that are still outstanding during the given time-frame in which the results are displayed.