1. Computers
  2. Display Drivers
  3. Graphics Cards
  4. Memory
  5. Motherboards
  6. Processors
  7. Software
  8. Storage
  9. Operating Systems

Facebook RSS Twitter Twitter Google Plus

Phoronix Test Suite

OpenBenchmarking Benchmarking Platform
Phoromatic Test Orchestration

Test Automation With PTS, OpenBenchmarking.org


Published on 09 May 2012 10:13 PM EDT
Written by Michael Larabel in Phoronix

Tonight at the Ubuntu Developer Summit I'm talking about the Phoronix Test Suite and OpenBenchmarking.org. Here's some of my notes for those unable to make it or were able to attend but would like additional information.

What Is The Phoronix Test Suite? An open-source test automation framework designed for benchmarking. The Phoronix Test Suite is designed to be extensible, multi-platform (any Linux distribution, Mac OS X, Solaris, BSDs, and even limited ports to Windows and Hurd and MINIX3), and easy-to-use while producing easily reproducible results. Everything from test download and installation to installed software/hardware reporting, test execution, and result reporting is fully-automated. The Phoronix Test Suite has been publicly available for four years now and builds upon the eight years of Linux benchmarking experience at Phoronix.com along with input from various tier-one vendors. There's over 200 test profiles available and more than 60 test suites while new, custom tests can be very easily introduced for public or private use. The Phoronix Test Suite isn't limited to solely quantitative results but can also handle image quality comparisons, pass/fail scenarios, and other abstract result types.

The Phoronix Test Suite can be easily used for automated testing in regression testing on a per-commit (or timed) basis, comparing hardware performance, comparing software performance, testing tuned vs. untuned systems, etc. The Phoronix Test Suite client is freely and openly available. There's commercial services offered for custom test development, software customizations, behind-the-firewall results storage, behind-the-firewall OpenBenchmarking.org, and other custom engineering work. The Phoronix Test Suite is already widely deployed amongst tier-one hardware vendors, universities, energy companies, financial institutions / proprietary trading firms, data centers, and much more.

What Is OpenBenchmarking.org? A complement to the Phoronix Test Suite that provides a centralized location for result storage, test profiles, and test suites. It's collaborative and allows new results to be easily shared, compared, and analyzed in different ways. OpenBenchmarking.org offers various features for result analytics, shopping assistance features, hardware indexes, etc. OpenBenchmarking.org provides as well a cloud-based package-management-like system for managing test profile versioning to ensure comparisons are properly binded to correct test versions, etc. OpenBenchmarking.org is basically a collaborative, crowd-sourced benchmarking platform that has been available since late 2010 and succeeds the former "PTS Global" implementation. There's also Phoromatic that will be offered as part of this for

Some Of The Recent and Upcoming Improvements:

- Compiler reporting improvements to show any switches/flags passed by any test profile (it's all fully automated, of course) to the C/C++/Fortran compilers when the program under test is being built (optimization flags, linked libraries, etc). There's also support for showing how the compiler being used in such comparisons is being built.

- When detecting disk comparisons, it now also highlights the various file-system mount options used for the disk partition under test. This is on top of all the reporting, system log capturing, and other safeguards the Phoronix Test Suite has long provided for ensuring sane, accurate, and reproducible testing. These reporting improvements basically just make the most relevant bits for any given result file become more easily to spot and compare rather than digging through the archived system logs.

- Likewise, now when detecting a CPU comparison is being conducted, a table showing the various CPU flags, cache sizes, and other CPU attributes is now prominently displayed beyond the other details and logs already captured.

- Lastly, when a GPU/graphics-driver comparison is detected, a complementary table appears that shows the differences in supported OpenGL extensions, GPGPU cores, and other GPU attributes to easily compare the technical features of each hardware product. More details on the CPU and GPU reporting enhancements here.

Some Of What's Coming Very Soon To OpenBenchmarking.org:

- Shopping-cart-like comparisons.

- Major search enhancements.

- Greater visibility into test profile operations.

- Phoromatic-over-OpenBenchmarking.org.

- Graphing and other interaction improvements.

Other Random Notes:

- There's more than 1.5 million computer hardware component configurations/metrics stored on OpenBenchmarking.org, more than 464,000 benchmarks were recorded on OpenBenchmarking.org, detailed statistics/compatibility information on more than 150,000 systems, and much more.

- The Phoronix Test Suite is extremely extensible and nearly any test can be added.

- The Phoronix Test Suite also supports modules, such as for system monitoring to automatically record the CPU temperature, system power consumption, memory usage, CPU usage, and other system vitals/sensors while testing is being done. From there the Phoronix Test Suite can automatically provide performance-per-Watt data and more.

- A long overdue and major update to PTS Desktop Live is going to be released around the same time as Phoronix Test Suite 4.0 with many exciting features. Details to be announced soon.

- Unlike the Canonical test products, the Phoronix Test Suite will work well on any Linux distribution or nearly any non-Linux operating system for that matter. Red Hat Enterprise Linux, Fedora, FreeBSD, and others all just work "out of the box" and will function and act the same way just as it does on Ubuntu Linux. This is very important for vendors concerned more than just being Ubuntu-specific.

- Due to the visibility of test results on Phoronix.com, there's frequent feedback from upstream software developers, hardware vendors, and other organizations to better evolve testing methods, etc. The Phoronix Test Suite is truly independent from any single hardware/software vendor.

- A test profile basically just consists of a few XML files and then a simple bash script for managing the test installation and execution. (The documentation right now is a bit limited in this area, but if anyone has any questions please contact us and happy to help out. Also happy to do custom engineering work or custom test profile development via the PTS commercial entity.)

- The Phoronix Test Suite software itself has minimal dependencies and will then dynamically install "external dependencies" based upon what's needed for a given test. The Phoronix Test Suite supports using the operating system's native package manager in these cases on nearly all Linux distributions.

- Test profiles can be easily modified in any manner to suit your needs. Ubuntu guys (as they have asked), you can easily modify test profiles to use the distribution packages where available rather than building the test itself from source.

- If any questions, feel free to ask via public or private channels!

About The Author
Michael Larabel is the principal author of Phoronix.com and founded the web-site in 2004 with a focus on enriching the Linux hardware experience and being the largest web-site devoted to Linux hardware reviews, particularly for products relevant to Linux gamers and enthusiasts but also commonly reviewing servers/workstations and embedded Linux devices. Michael has written more than 10,000 articles covering the state of Linux hardware support, Linux performance, graphics hardware drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org automated testing software. He can be followed via and or contacted via .
Latest Articles & Reviews
  1. Trying Out The Modern Linux Desktops With 4 Monitors + AMD/NVIDIA Graphics
  2. Turning A Basement Into A Big Linux Server Room
  3. NVIDIA's $1000+ GeForce GTX TITAN X Delivers Maximum Linux Performance
  4. OS X 10.10 vs. Ubuntu 15.04 vs. Fedora 21 Tests: Linux Sweeps The Board
  5. The New Place Where Linux Code Is Constantly Being Benchmarked
  6. 18-GPU NVIDIA/AMD Linux Comparison Of BioShock: Infinite
Latest Linux News
  1. Firefox 37 Coming Today With Heartbeat, HTTPS Bing
  2. OpenIndiana 2015.03 Updates Its Solaris/Illumos Environment
  3. GNOME 3.16 SDK Runtime Now Available
  4. Initial Intel Braxton Support Might Come To Linux 4.1
  5. Why KDE's KWin Doesn't Integrate Weston/QtCompositor For Wayland Support
  6. Clang Now Supports Targeting The NaCl OS
  7. PC-BSD Updates Its Lumina Desktop (v0.8.3)
  8. Fedora 22 Alpha Now Available For AArch64 & PowerPC64
  9. Systemd Developers Did NOT Fork The Linux Kernel
  10. PulseAudio 7.0 To Enable LFE Remixing By Default
Most Viewed News This Week
  1. Introducing The Library Operating System For Linux
  2. Improved OpenCL Support For Blender's Cycles Renderer
  3. Allwinner Continues Jerking Around The Open-Source Community
  4. Open-Source Driver Fans Will Love NVIDIA's New OpenGL Demo
  5. GNOME 3.16 Released: It's Their Best Release Yet
  6. Systemd Change Allows For Stateless Systems With Tmpfs
  7. Ubuntu 15.04 Final Beta Released
  8. Nuclide: Facebook's New Unified IDE