1. Computers
  2. Display Drivers
  3. Graphics Cards
  4. Memory
  5. Motherboards
  6. Processors
  7. Software
  8. Storage
  9. Operating Systems


Facebook RSS Twitter Twitter Google Plus


Phoronix Test Suite

OpenBenchmarking.org

Bringing Up Advancements In Compiler Testing

Phoronix

Published on 07 March 2012 11:32 AM EST
Written by Michael Larabel in Phoronix
19 Comments

For those interested in compiler testing and tuning performance, there's some notable advancements to share that will benefit the community.

With Phoronix Test Suite 3.8-Bygland, which will be officially released this month, there are some last-minute changes being pushed for this quarterly release cycle. There's already a number of new features in Phoronix Test Suite 3.8, but being pushed in recent days has been the start of advancements to enhance the compiler testing abilities of the Phoronix Test Suite and its reporting of compiler options and other system settings.

One of the Phoronix Test Suite commercial customers is sponsoring the development of several new features for Phoronix Test Suite 3.8 and the major Phoronix Test Suite 4.0 release later in the year. One of their big areas of interest is with compiler-based testing, and fortunately they're kind enough to let the sponsored work be pushed directly into the open-source/community Phoronix Test Suite code-base. What's landed in the Git repository already for the forthcoming Phoronix Test Suite 3.8 is automated reporting of compiler options.

Through a compiler-masking method, compiler options are being seamlessly monitored and recorded by the Phoronix Test Suite upon each test installation. That information is then later automatically analyzed looking for compiler differences between test runs and to then pass the relevant information onto pts_Graph for rendering. Here's the brief compiler testing description from the Phoronix Test Suite documentation.
A majority of the test profiles provided by OpenBenchmarking.org to the Phoronix Test Suite are source-based tests. Relying upon the upstream source-code for each program under test allows for the tests to be easily brought to new platforms and architectures, avoids any out-of-tree / non-default packaging differences by different distributions and operating systems, and to allow the Phoronix Test Suite user to easily test new compilers and/or compiler options. For the source-based tests, the Phoronix Test Suite relies upon a compiler (e.g. GCC, LLVM/Clang, Sun Studio, Open64, et al) being present on the system under test. The Phoronix Test Suite does respect CC/CXX environment variables and test profiles are expected to honor CFLAGS/CXXFLAGS and other compiler settings.

As of Phoronix Test Suite 3.8, a compiler masking method is utilized for logging compiler options and other settings with each test profile installation. Prior to installing an open-source test, the Phoronix Test Suite determines the intended compiler to be used based upon the pre-set environment variables or the pre-set compiler(s) within the PATH environment variable. The Phoronix Test Suite then masks the compiler to ensure that any options/flags submitted to the compiler are first passed through pts-core so that they can be logged for later use, then is linked to the original, intended compiler. Additionally, other compiler binary names of the same type are blacklisted to prevent their un-intended use (i.e. if a test profile has hard-coded gcc in its build script, but clang is set as the compiler via CC, a sym-link will automatically be created from gcc to the masked clang for the duration of its test installation).

The logged compiler data is then used by the Phoronix Test Suite following the test execution process for automated result analysis. If there is a detected change in compiler settings, the differences are automatically reported to the test result graphs. Additionally, key compiler information (e.g. the compiler optimization level and key libraries that were linked at compile-time) is reported as a footnote on result graphs. The Phoronix Test Suite handles all of this in a fully automated manner.

Below are a few sample graphs just intended to highlight the new compiler reporting that's handled automatically by the Phoronix Test Suite (the results are just sample data locally).

Bringing Up Advancements In Compiler Testing

From this first example, it was a simple run of just running the same test multiple times. As you can see, the new addition to 3.8-Bygland when running this test profile is that some of the compiler information is now auto-reported as a footnote. It shows that by default OpenMP is being used and the compiler optimization level (-O2). This is especially helpful for new users that may not be familiar with the specifics of a test profile. This is all done automatically without requiring any test profile updates or other changes. As has always been the case, the compiler version and other logs (e.g. cc -v) is already captured and available from the system information area.

Bringing Up Advancements In Compiler Testing

With the second example of the same test profile, the Phoronix Test Suite determined that the compiler's optimization level was being changed between runs. As a result, on each of the result bars it now shows the varying information (it's also checked to verify that the result identifier doesn't contain this information, so it doesn't appear redundant). As a foot-note it still indicates that OpenMP is being used by the test, since that was not changed between test runs.

Bringing Up Advancements In Compiler Testing

In this example, the Phoronix Test Suite detected that the compiler was being changed out (via CC) when the test was being re-installed.

These compiler reporting improvements will also be visible on the graphs generated by OpenBenchmarking.org in the coming weeks. The version of pts-core running OpenBenchmarking.org simply needs to synced again to benefit from this work. This latest work is found in the Git code-base right now and will be featured in Phoronix Test Suite 3.8-Bygland Milestone 3 to be released in the next week. The official 3.8 release is still expected by month's end to complete the quarter.

Stay tuned for more reporting advancements plus other features. Additionally, any questions or feedback by the community is welcome on how this work can be expanded or be made more beneficial.

About The Author
Michael Larabel is the principal author of Phoronix.com and founded the web-site in 2004 with a focus on enriching the Linux hardware experience and being the largest web-site devoted to Linux hardware reviews, particularly for products relevant to Linux gamers and enthusiasts but also commonly reviewing servers/workstations and embedded Linux devices. Michael has written more than 10,000 articles covering the state of Linux hardware support, Linux performance, graphics hardware drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org automated testing software. He can be followed via and or contacted via .
Latest Linux Hardware Reviews
  1. Even With Re-Clocking, Nouveau Remains Behind NVIDIA's Proprietary Linux Driver
  2. The Power Consumption & Efficiency Of Open-Source GPU Drivers
  3. AMD R600g/RadeonSI Performance On Linux 3.16 With Mesa 10.3-devel
  4. Intel Pentium G3258 On Linux
Latest Linux Articles
  1. AMD Catalyst 14.6 Does Slightly Better With APITest OpenGL Tests
  2. Updated Source Engine Benchmarks On The Latest AMD/NVIDIA Linux Drivers
  3. Nouveau vs. Radeon vs. Intel Tests On Linux 3.16, Mesa 10.3-devel
  4. KVM Benchmarks On Ubuntu 14.10
Latest Linux News
  1. Belkin's WRT54G Router Successor Is Crap On The Software Front So Far
  2. QEMU 2.1 Officially Released
  3. XBMC Is Being Renamed To "Kodi"
  4. SteamOS Beta 126 Better Handles XBMC
  5. Ubuntu 14.10 Alpha 2 Released
  6. KDE 4.14 Release Candidate Ships
  7. Drivers & Drama Dominated Linux Talk In July
  8. Fedora Assembles A Security Team
  9. AMD Launches The A10-7800, The 65 Watt Kaveri
  10. Builder: A New Development IDE Being Built For GNOME
Latest Forum Discussions
  1. AMD Launches The A10-7800, The 65 Watt Kaveri
  2. Grand Theft Auto Running On Direct3D Natively On Linux Shows Gallium3D Potential
  3. Updated and Optimized Ubuntu Free Graphics Drivers
  4. Linus Torvalds On GCC 4.9: Pure & Utter Crap
  5. Updated graphics drivers for Ubuntu 12.04 Precise LTS
  6. Debian + radeonsi
  7. AMD Publishes Open-Source Linux HSA Kernel Driver
  8. Open-source drivers on ATI R7 260X