1. Computers
  2. Display Drivers
  3. Graphics Cards
  4. Memory
  5. Motherboards
  6. Processors
  7. Software
  8. Storage
  9. Operating Systems


Facebook RSS Twitter Twitter Google Plus


Phoronix Test Suite

OpenBenchmarking.org

Bye-Bye Test Profiles & Suites; Hi OpenBenchmarking.org

Phoronix

Published on 07 December 2010 12:23 PM EST
Written by Michael Larabel in Phoronix
Comment On This Article

For those not tracking the Phorogit code of the Phoronix Test Suite, all of the test profiles and test suites living within our open-source benchmarking software have been removed. All 130+ test profiles and ~50 test suites have been dropped, as they're no longer living within the benchmarking package.

Why? They're moving to OpenBenchmarking.org. As talked about in late October about how Linux benchmarking will change, with Phoronix Test Suite 3.0 "Iveland" we are are introducing OpenBenchmarking.org as the next-generation Phoronix Global implementation that far trumps the current capabilities. The OpenBenchmarking.org infrastructure provides public and private storage of test result data (including system logs and benchmark outputs) and effective collaboration tools for sharing results and efficiently comparing multiple test result sets. OpenBenchmarking.org also provides a package management system for making accessible new, updated, and third-party test profiles / suites to the users of the Phoronix Test Suite. Plus a lot more.

Bye-Bye Test Profiles & Suites; Hi OpenBenchmarking.org


With OpenBenchmarking.org and Phoronix Test Suite 3.0+, the test profiles and suites live out in the "cloud" and are pulled down automatically in a manner like a package management system. This provides a number of benefits, including:

- You no longer need to wait for a new version of the Phoronix Test Suite to be released to gain access to new or updated test profiles in an easy-to-use manner. Now you can instantly tap into new/updated test profiles in a seamless manner upon their upload to OpenBenchmarking.org. As long as the test profiles (or test suites) don't depend upon capabilities that require an updated version of the Phoronix Test Suite (pts-core will automatically resolve incompatible tests/suites), you can opt to install/upgrade.

- Upgrading your Phoronix Test Suite client no longer requires you to use the new/updated test profiles and suites. This is great especially in the enterprise environment where you may be tracking the performance of hardware across product generations or benchmarking the latest software stack on a daily basis over a long term (i.e. with Phoromatic and using Phoromatic Tracker to benchmark the Linux kernel daily) and would prefer to stick with the same exact test version for a number of months or even years. Now you can still upgrade the Phoronix Test Suite to gain access to any new pts-core features or other improvements while still binding your testing to a particular test profile or suite version.

- Multiple versions of a test/suite can be installed simultaneously.

- If running a performance comparison of your system against someone else's system, you can ensure now that you have the same exact version of the test(s) installed that are used in the comparison, regardless of the Phoronix Test Suite release you are using.

- Moving to OpenBenchmarking.org ensures that third-parties can easily publish their own test profiles while gaining visibility. Users will be allowed to upload their own test profiles and suites (you can also build your own test suites easily through the web-interface!) to OpenBenchmarking.org and they will be treated in the same way as the "official" test profiles and suites. Users will not have to jump through any hurdles or manually setup any files no longer to utilize third-party resources. When you upload results to share on OpenBenchmarking.org using one of these custom/third-party tests, the Phoronix Test Suite can automatically fetch the needed profiles/suites automatically too!

- Plus some other features being explored that aren't yet ready to be announced ;)

Bye-Bye Test Profiles & Suites; Hi OpenBenchmarking.org


Right now in the latest Git code there are still a few bugs and issues being worked out, but there will be a bigger call for testing beginning with the Phoronix Test Suite 3.0 Alpha 4 release later in the month.

Below are some notes on the technical implementation of the OpenBenchmarking.org package management system implementation in the Phoronix Test Suite.

- With Phoronix Test Suite 3.0, you still can run tests just like you have done since Phoronix Test Suite 0.1. Running a command like phoronix-test-suite benchmark apache will work just fine. The official Phoronix Test Suite test/suite repository (simply named "pts") is enabled by default in Iveland as the primary package source. Internally, this evaluates to the equivalent of running phoronix-test-suite benchmark pts/apache-1.1.0. The Phoronix Test Suite scans the available repositories for the matching name and if no version for the test profile is supplied, it will default to the latest. Though if you want to run an older version of a test, you can simply supply it in the phoronix-test-suite benchmark repository/test-version format.

- For third-party repositories you can add the repository to be part of your default index (thereby no longer needing to supply the repository/ portion, unless the same test name happens to exist in pts/ or a higher-level repository). Otherwise you can always install/upgrade a test or suite by simply append the repository prefix (i.e. phoronix-test-suite benchmark some-hardware-vendor/new-opencl-test).

- This same string evaluation also occurs within result files and test suites. This allows you to either define a particular test profile version to use when running a test suite or to always just use the most recent test profile version as its default. This also means all test profiles and suites are treated equally and you can even mix and match tests/suites from different OpenBenchmarking.org repositories.

- For those enterprise users running the Phoronix Test Suite on an intranet where the test nodes do not have access to the Internet, it will be possible to copy over the OpenBenchmarking.org files and import them locally. There is also HTTPS communication support.

Plus there's many more features on the horizon for the package system along with OpenBenchmarking.org (result collaboration, interactive and embed-able graphs, etc) and PTS Iveland in general. Stay tuned for more information and we invite your feedback. As said earlier, "OpenBenchmarking.org is a new online component for benchmarking that will launch in conjunction with Phoronix Test Suite 3.0 "Iveland" early next year. While it's not even out publicly yet, I expect OpenBenchmarking.org to eventually become bigger -- both in terms of traffic and even greater industry importance -- than Phoronix.com itself." The official release is coming in Q1'2011.

About The Author
Michael Larabel is the principal author of Phoronix.com and founded the web-site in 2004 with a focus on enriching the Linux hardware experience and being the largest web-site devoted to Linux hardware reviews, particularly for products relevant to Linux gamers and enthusiasts but also commonly reviewing servers/workstations and embedded Linux devices. Michael has written more than 10,000 articles covering the state of Linux hardware support, Linux performance, graphics hardware drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org automated testing software. He can be followed via and or contacted via .
Latest Linux Hardware Reviews
  1. NVIDIA GeForce GTX 980: The Best GPU For Linux Gamers
  2. ROCCAT LUA: A Linux-Friendly Gaming Mouse
  3. Cheetah Mounts: The Affordable Way To Put Your TV On The Wall
  4. Scythe Mugen MAX
Latest Linux Articles
  1. Preview: Radeon Gallium3D Performance For CS:GO On Linux
  2. XWayland Linux Gaming Performance With GNOME Wayland On Fedora 21
  3. EXT4/Btrfs/XFS/F2FS Benchmarks On Linux 3.17
  4. Fedora 21 Alpha First Impressions: It's Great
Latest Linux News
  1. KDE Plasma 5.1 Now In Beta
  2. Systemd & Debian Were Most Popular In September
  3. Microsoft Announces... Windows 10 With A Start Menu
  4. Borderlands 2 Launches On Steam For Linux
  5. Debian Jessie Might Get Rid Of The kFreeBSD Port
  6. Fedora Might Try A New Scheduling Strategy For Its Releases
  7. AMD's Catalyst Working On A GLSL Shader Cache
  8. OpenMP 4.0 Offloading Is Closer For GCC 5
  9. Wayland Presentation Extension Added To Weston
  10. Intel Skylake Support Rolls Out To Mesa's DRM
Latest Forum Discussions
  1. New Group Calls For Boycotting Systemd
  2. FSF Issues Statement On Shellshock Bash Vulnerability
  3. Updated and Optimized Ubuntu Free Graphics Drivers
  4. X.Org Women Outreach Program Only Turns Up Two Applicants So Far
  5. X.Org Is Looking For Some Female Help
  6. Advertisements On Phoronix
  7. NVIDIA Alerts Nouveau: They're Starting To Sign/Validate GPU Firmware Images
  8. Take the Steam Survey results with a grain of salt. It is flawed.