Phoronix Test Suite 6.0 Delivers New Result/Phoromatic UI, New Modules

After a half-year of development, I'm ecstatic to announce this morning the release of Phoronix Test Suite 6.0 (codenamed "Hammerfest"). Phoronix Test Suite 6.0 is by far the most significant release ever done of our open-source, cross-platform automated benchmarking software and framework since the release of Phoronix Test Suite 1.0 seven years ago.

There have been a ton of changes to Phoronix Test Suite 6.0-Hammerfest that are both internal to benefit new features coming later in Phoronix Test Suite 6.x as well as many user-facing improvements for those relying upon our GPL-licensed benchmarking and test automation software. Phoronix Test Suite 6.0 is available for immediate download this morning via Phoronix-Test-Suite.com or via Phoronix-Test-Suite on GitHub.

New Phoromatic UI

Since the open-source Phoromatic Server was merged into the mainline Phoronix Test Suite code-base in Phoronix Test Suite 5, our test orchestration and management component has seen many significant additions throughout the 5.x releases. With Phoronix Test Suite 6.0, many new features continue to be added to the built-in Phoromatic server and client. One of the visible changes with Phoronix Test Suite 6.0 is that the site layout has been revised and there's initial support for mobile devices. The theme has also been updated to be more cohesive with the new result viewer and Phoronix Test Suite site layout.

New Result Viewer

The local Phoronix Test Suite result viewer has been completely rewritten. Previously the viewer relied upon XSL for styling the result viewer, but with the browser support for it beginning to sway, the new viewer has been written to just utilize HTML and JavaScript for processing the XML result data. This new viewer has been tested on all major web-browsers and is working out much better. Beyond the internal changes, the default theme of the result viewer has been updated to better jive with the new Phoronix-Test-Suite.com site that rolled out some weeks ago.

More Phoromatic Additions

The Phoromatic Server now has a new settings option for auto-authenticating new clients to automatically approve new systems attempting to connect to a Phoromatic account. The Phoromatic Server now also has support for exporting results as XML directly from the web interface, alongside the PDF and CSV result exporting. The Phoromatic Server also now has a test profiles page for showing basic test information along with results associated with a given test, similar to functionality found on OpenBenchmarking.org.

Low-Level Improvements

A lot of the internal, low-level code of the Phoronix Test Suite's pts-core was significantly reworked. In fact, a lot of code that dates back to Phoronix Test Suite pre-1.0 was rewritten to take advantage of modern PTS interfaces and new PHP language features. This rework yields significant speed improvements (covered in the next section) while allowing for new functionality to be more easily implemented thanks to the cleaner and more streamlined code-base. When the Phoronix Test Suite was originally written, it wasn't designed around the yet-to-be-conceived OpenBenchmarking.org and Phoromatic while the new code does better accommodate these new components and other yet-to-be-announced features. In particular, Phoronix Test Suite 6.0 should respond much better for doing big data analytics on massive performance result sets.

Greater Performance

While the pre-Hammerfest code works fine for all users running the Phoronix Test Suite individually and just running as many benchmarks to your delight on a system, the low-level re-work was driven in part for ensuring greater efficiency out of companies running their own internal Phoromatic or OpenBenchmarking.org servers. Phoronix Test Suite 6.0 may not seem much faster when you're just running a couple benchmarks on the system, but when rendering thousands of results in real-time for display, there's a dramatic speed-up.

When using the Phoronix Test Suite's own self-hosting benchmarking (phoronix-test-suite debug-self-test or phoronix-test-suite benchmark pts-self-test for it in the self-hosting test profile mode), the performance improvements of Phoronix Test Suite 6.0 are quite clear. The total time of the test that stresses all of the hot code-paths of the software dropped from 617 seconds to complete to just over 200 seconds... In other words, about three times faster all through this Phoronix Test Suite update with the same PHP stack.

One of the big focuses with the low-level re-work for Phoronix Test Suite 6.0 was on improving the graph rendering process: from the parsing of the result XML to generating the respective SVG or PNG graphs and then passing them off to the local result viewer, PDF result file, or dynamically serving them on OpenBenchmarking.org or Phoromatic. Thanks to the successful rework, the graph rendering process is significantly faster. With all of the internal tests done, this graph rendering process is about 4x faster than Phoronix Test Suite 5.8! If you're just running the Phoronix Test Suite locally you may not notice any difference when it already took less than one second, but on OpenBenchmarking.org where millions of graphs are rendered each month, it's a big impact on system resources. This is also of much benefit to our LinuxBenchmarking.com reference tracker where now it takes significantly less time to dynamically pull up a lot of historical benchmark results. For Phoromatic users, it will also mean much less time is needed when running large comparisons on existing results or looking at the performance of your systems under test for several months/years.

Besides being much faster, pts-core uses significantly less memory during these heavy tasks. With Phoronix Test Suite 5.8.1 the peak memory usage came up to almost 1GB when rendering thousands of results at once while now with Phoronix Test Suite 6.0 the peak memory usage is around a 4x improvement. Again, this will be most dramatic for Phoromatic and OpenBenchmarking.org users.

Lastly, another low-level improvement made was a big rework that happened within the graphing code (pts_Graph). The changes will allow for our graphs to be richer and more extensible moving forward, but an immediate win -- besides partially allowing the faster performance -- is that pts_Graph takes care of generating more-optimized SVG graphs. Common graph types are now roughly half the size as with previous versions of the Phoronix Test Suite. PNG/JPEG graphs continue to be supported for integration into PDF files as well as for supporting legacy web browsers.

Reworked Sensor Monitoring

There was a significant rework to how the Phodevi's (Phoronix Device Interface) sensor monitoring works. The new sensor framework is now properly multi-threaded, much more extensible, and now also supports the dynamic creation of sensors in cases of desiring to monitor CPU usage or temperatures on a per-core basis rather than just the overall package or when looking to otherwise sub-divide the sensor data. The Phoronix Test Suite sensors behave in the same way via the MONITOR= environment variable and phoronix-test-suite system-sensors.

Phodevi Support Additions

There's now support for the GPU thermal monitoring on more hardware (including the NVIDIA Jetson TX1), version detection for the X.Org Server running as non-root on Debian, NVMe solid-state storage detection, AMDGPU DRM detection, BcacheFS super-block detection, and more.

Linux Perf Subsystem Module

There's a new module called linux_perf that supports reporting various Linux perf subsystem performance counters on a per-test-run basis. This module is automatically loaded when the LINUX_PERF=1 environment variable is set prior to running the Phoronix Test Suite.

Performance-Per-Dollar Module

Another new Phoronix Test Suite module provides automated performance-per-dollar graphs based upon the user-supplied price information for the component/system under test. It's an easy way of running a cost analysis to complement the raw performance figures. This module is enabled automatically via the COST_PERF_PER_DOLLAR= environment variable and setting it to the dollar value desired for the current test run.

Plus More!

There's also various improvements in other areas of the Phoronix Test Suite, Windows support updates, DragonFlyBSD external dependency handling via DPorts pkg rather than pkgsrc, pass/fail testing improvements, the estimated date/time for test completion is now shown on the CLI besides the estimated time remaining, better support for stateless systems, and other smaller changes.

Looking ahead to 2016, the next-generation version of OpenBenchmarking.org remains under development and should be public within the next few months. Initial plans for Phoronix Test Suite 6.2-Gamvik include more Phoromatic features being merged, new analytic features, continued refinements to the new user interfaces, and other new testing possibilities. Phoronix Test Suite 6.2 is planned for release in Q1'2016.

Phoronix Test Suite 6.0 can be downloaded from Phoronix-Test-Suite.com or via Phoronix-Test-Suite on GitHub. Those in need of commercial support, custom engineering, or other professional services are asked to contact us.