1. Computers
  2. Display Drivers
  3. Graphics Cards
  4. Memory
  5. Motherboards
  6. Processors
  7. Software
  8. Storage
  9. Operating Systems


Facebook RSS Twitter Twitter Google Plus


Phoronix Test Suite

OpenBenchmarking.org

Test Automation With PTS, OpenBenchmarking.org

Phoronix

Published on 09 May 2012 10:13 PM EDT
Written by Michael Larabel in Phoronix
5 Comments

Tonight at the Ubuntu Developer Summit I'm talking about the Phoronix Test Suite and OpenBenchmarking.org. Here's some of my notes for those unable to make it or were able to attend but would like additional information.

What Is The Phoronix Test Suite? An open-source test automation framework designed for benchmarking. The Phoronix Test Suite is designed to be extensible, multi-platform (any Linux distribution, Mac OS X, Solaris, BSDs, and even limited ports to Windows and Hurd and MINIX3), and easy-to-use while producing easily reproducible results. Everything from test download and installation to installed software/hardware reporting, test execution, and result reporting is fully-automated. The Phoronix Test Suite has been publicly available for four years now and builds upon the eight years of Linux benchmarking experience at Phoronix.com along with input from various tier-one vendors. There's over 200 test profiles available and more than 60 test suites while new, custom tests can be very easily introduced for public or private use. The Phoronix Test Suite isn't limited to solely quantitative results but can also handle image quality comparisons, pass/fail scenarios, and other abstract result types.

The Phoronix Test Suite can be easily used for automated testing in regression testing on a per-commit (or timed) basis, comparing hardware performance, comparing software performance, testing tuned vs. untuned systems, etc. The Phoronix Test Suite client is freely and openly available. There's commercial services offered for custom test development, software customizations, behind-the-firewall results storage, behind-the-firewall OpenBenchmarking.org, and other custom engineering work. The Phoronix Test Suite is already widely deployed amongst tier-one hardware vendors, universities, energy companies, financial institutions / proprietary trading firms, data centers, and much more.

What Is OpenBenchmarking.org? A complement to the Phoronix Test Suite that provides a centralized location for result storage, test profiles, and test suites. It's collaborative and allows new results to be easily shared, compared, and analyzed in different ways. OpenBenchmarking.org offers various features for result analytics, shopping assistance features, hardware indexes, etc. OpenBenchmarking.org provides as well a cloud-based package-management-like system for managing test profile versioning to ensure comparisons are properly binded to correct test versions, etc. OpenBenchmarking.org is basically a collaborative, crowd-sourced benchmarking platform that has been available since late 2010 and succeeds the former "PTS Global" implementation. There's also Phoromatic that will be offered as part of this for

Some Of The Recent and Upcoming Improvements:

- Compiler reporting improvements to show any switches/flags passed by any test profile (it's all fully automated, of course) to the C/C++/Fortran compilers when the program under test is being built (optimization flags, linked libraries, etc). There's also support for showing how the compiler being used in such comparisons is being built.

- When detecting disk comparisons, it now also highlights the various file-system mount options used for the disk partition under test. This is on top of all the reporting, system log capturing, and other safeguards the Phoronix Test Suite has long provided for ensuring sane, accurate, and reproducible testing. These reporting improvements basically just make the most relevant bits for any given result file become more easily to spot and compare rather than digging through the archived system logs.

- Likewise, now when detecting a CPU comparison is being conducted, a table showing the various CPU flags, cache sizes, and other CPU attributes is now prominently displayed beyond the other details and logs already captured.

- Lastly, when a GPU/graphics-driver comparison is detected, a complementary table appears that shows the differences in supported OpenGL extensions, GPGPU cores, and other GPU attributes to easily compare the technical features of each hardware product. More details on the CPU and GPU reporting enhancements here.

Some Of What's Coming Very Soon To OpenBenchmarking.org:

- Shopping-cart-like comparisons.

- Major search enhancements.

- Greater visibility into test profile operations.

- Phoromatic-over-OpenBenchmarking.org.

- Graphing and other interaction improvements.

Other Random Notes:

- There's more than 1.5 million computer hardware component configurations/metrics stored on OpenBenchmarking.org, more than 464,000 benchmarks were recorded on OpenBenchmarking.org, detailed statistics/compatibility information on more than 150,000 systems, and much more.

- The Phoronix Test Suite is extremely extensible and nearly any test can be added.

- The Phoronix Test Suite also supports modules, such as for system monitoring to automatically record the CPU temperature, system power consumption, memory usage, CPU usage, and other system vitals/sensors while testing is being done. From there the Phoronix Test Suite can automatically provide performance-per-Watt data and more.

- A long overdue and major update to PTS Desktop Live is going to be released around the same time as Phoronix Test Suite 4.0 with many exciting features. Details to be announced soon.

- Unlike the Canonical test products, the Phoronix Test Suite will work well on any Linux distribution or nearly any non-Linux operating system for that matter. Red Hat Enterprise Linux, Fedora, FreeBSD, and others all just work "out of the box" and will function and act the same way just as it does on Ubuntu Linux. This is very important for vendors concerned more than just being Ubuntu-specific.

- Due to the visibility of test results on Phoronix.com, there's frequent feedback from upstream software developers, hardware vendors, and other organizations to better evolve testing methods, etc. The Phoronix Test Suite is truly independent from any single hardware/software vendor.

- A test profile basically just consists of a few XML files and then a simple bash script for managing the test installation and execution. (The documentation right now is a bit limited in this area, but if anyone has any questions please contact us and happy to help out. Also happy to do custom engineering work or custom test profile development via the PTS commercial entity.)

- The Phoronix Test Suite software itself has minimal dependencies and will then dynamically install "external dependencies" based upon what's needed for a given test. The Phoronix Test Suite supports using the operating system's native package manager in these cases on nearly all Linux distributions.

- Test profiles can be easily modified in any manner to suit your needs. Ubuntu guys (as they have asked), you can easily modify test profiles to use the distribution packages where available rather than building the test itself from source.

- If any questions, feel free to ask via public or private channels!

About The Author
Michael Larabel is the principal author of Phoronix.com and founded the web-site in 2004 with a focus on enriching the Linux hardware experience and being the largest web-site devoted to Linux hardware reviews, particularly for products relevant to Linux gamers and enthusiasts but also commonly reviewing servers/workstations and embedded Linux devices. Michael has written more than 10,000 articles covering the state of Linux hardware support, Linux performance, graphics hardware drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org automated testing software. He can be followed via and or contacted via .
Latest Linux Hardware Reviews
  1. NVIDIA GeForce GTX 980: The Best GPU For Linux Gamers
  2. ROCCAT LUA: A Linux-Friendly Gaming Mouse
  3. Cheetah Mounts: The Affordable Way To Put Your TV On The Wall
  4. Scythe Mugen MAX
Latest Linux Articles
  1. Preview: Radeon Gallium3D Performance For CS:GO On Linux
  2. XWayland Linux Gaming Performance With GNOME Wayland On Fedora 21
  3. EXT4/Btrfs/XFS/F2FS Benchmarks On Linux 3.17
  4. Fedora 21 Alpha First Impressions: It's Great
Latest Linux News
  1. Debian Jessie Might Get Rid Of The kFreeBSD Port
  2. Fedora Might Try A New Scheduling Strategy For Its Releases
  3. AMD's Catalyst Working On A GLSL Shader Cache
  4. OpenMP 4.0 Offloading Is Closer For GCC 5
  5. Wayland Presentation Extension Added To Weston
  6. Intel Skylake Support Rolls Out To Mesa's DRM
  7. VA-API's Libva 1.4.0 Brings VP8 Encoding Support
  8. Operating System U Fails To Live Up To Its Goals
  9. AMD Catalyst 14.9 Officially Released For Linux
  10. Nouveau Memory Re-Clocking Comes For More NVIDIA GPUs
Latest Forum Discussions
  1. NVIDIA Alerts Nouveau: They're Starting To Sign/Validate GPU Firmware Images
  2. New AMD Catalyst drivers out today
  3. Updated and Optimized Ubuntu Free Graphics Drivers
  4. Take the Steam Survey results with a grain of salt. It is flawed.
  5. FSF Issues Statement On Shellshock Bash Vulnerability
  6. AMD Wants To Know What's Wrong With Catalyst
  7. New Group Calls For Boycotting Systemd
  8. Counter-Strike: Global Offensive NVIDIA/AMD Benchmarks On Linux