1. Computers
  2. Display Drivers
  3. Graphics Cards
  4. Memory
  5. Motherboards
  6. Processors
  7. Software
  8. Storage
  9. Operating Systems


Facebook RSS Twitter Twitter Google Plus


Phoronix Test Suite

OpenBenchmarking.org

Bringing Up Advancements In Compiler Testing

Phoronix

Published on 07 March 2012 11:32 AM EST
Written by Michael Larabel in Phoronix
19 Comments

For those interested in compiler testing and tuning performance, there's some notable advancements to share that will benefit the community.

With Phoronix Test Suite 3.8-Bygland, which will be officially released this month, there are some last-minute changes being pushed for this quarterly release cycle. There's already a number of new features in Phoronix Test Suite 3.8, but being pushed in recent days has been the start of advancements to enhance the compiler testing abilities of the Phoronix Test Suite and its reporting of compiler options and other system settings.

One of the Phoronix Test Suite commercial customers is sponsoring the development of several new features for Phoronix Test Suite 3.8 and the major Phoronix Test Suite 4.0 release later in the year. One of their big areas of interest is with compiler-based testing, and fortunately they're kind enough to let the sponsored work be pushed directly into the open-source/community Phoronix Test Suite code-base. What's landed in the Git repository already for the forthcoming Phoronix Test Suite 3.8 is automated reporting of compiler options.

Through a compiler-masking method, compiler options are being seamlessly monitored and recorded by the Phoronix Test Suite upon each test installation. That information is then later automatically analyzed looking for compiler differences between test runs and to then pass the relevant information onto pts_Graph for rendering. Here's the brief compiler testing description from the Phoronix Test Suite documentation.
A majority of the test profiles provided by OpenBenchmarking.org to the Phoronix Test Suite are source-based tests. Relying upon the upstream source-code for each program under test allows for the tests to be easily brought to new platforms and architectures, avoids any out-of-tree / non-default packaging differences by different distributions and operating systems, and to allow the Phoronix Test Suite user to easily test new compilers and/or compiler options. For the source-based tests, the Phoronix Test Suite relies upon a compiler (e.g. GCC, LLVM/Clang, Sun Studio, Open64, et al) being present on the system under test. The Phoronix Test Suite does respect CC/CXX environment variables and test profiles are expected to honor CFLAGS/CXXFLAGS and other compiler settings.

As of Phoronix Test Suite 3.8, a compiler masking method is utilized for logging compiler options and other settings with each test profile installation. Prior to installing an open-source test, the Phoronix Test Suite determines the intended compiler to be used based upon the pre-set environment variables or the pre-set compiler(s) within the PATH environment variable. The Phoronix Test Suite then masks the compiler to ensure that any options/flags submitted to the compiler are first passed through pts-core so that they can be logged for later use, then is linked to the original, intended compiler. Additionally, other compiler binary names of the same type are blacklisted to prevent their un-intended use (i.e. if a test profile has hard-coded gcc in its build script, but clang is set as the compiler via CC, a sym-link will automatically be created from gcc to the masked clang for the duration of its test installation).

The logged compiler data is then used by the Phoronix Test Suite following the test execution process for automated result analysis. If there is a detected change in compiler settings, the differences are automatically reported to the test result graphs. Additionally, key compiler information (e.g. the compiler optimization level and key libraries that were linked at compile-time) is reported as a footnote on result graphs. The Phoronix Test Suite handles all of this in a fully automated manner.

Below are a few sample graphs just intended to highlight the new compiler reporting that's handled automatically by the Phoronix Test Suite (the results are just sample data locally).

Bringing Up Advancements In Compiler Testing

From this first example, it was a simple run of just running the same test multiple times. As you can see, the new addition to 3.8-Bygland when running this test profile is that some of the compiler information is now auto-reported as a footnote. It shows that by default OpenMP is being used and the compiler optimization level (-O2). This is especially helpful for new users that may not be familiar with the specifics of a test profile. This is all done automatically without requiring any test profile updates or other changes. As has always been the case, the compiler version and other logs (e.g. cc -v) is already captured and available from the system information area.

Bringing Up Advancements In Compiler Testing

With the second example of the same test profile, the Phoronix Test Suite determined that the compiler's optimization level was being changed between runs. As a result, on each of the result bars it now shows the varying information (it's also checked to verify that the result identifier doesn't contain this information, so it doesn't appear redundant). As a foot-note it still indicates that OpenMP is being used by the test, since that was not changed between test runs.

Bringing Up Advancements In Compiler Testing

In this example, the Phoronix Test Suite detected that the compiler was being changed out (via CC) when the test was being re-installed.

These compiler reporting improvements will also be visible on the graphs generated by OpenBenchmarking.org in the coming weeks. The version of pts-core running OpenBenchmarking.org simply needs to synced again to benefit from this work. This latest work is found in the Git code-base right now and will be featured in Phoronix Test Suite 3.8-Bygland Milestone 3 to be released in the next week. The official 3.8 release is still expected by month's end to complete the quarter.

Stay tuned for more reporting advancements plus other features. Additionally, any questions or feedback by the community is welcome on how this work can be expanded or be made more beneficial.

About The Author
Michael Larabel is the principal author of Phoronix.com and founded the web-site in 2004 with a focus on enriching the Linux hardware experience and being the largest web-site devoted to Linux hardware reviews, particularly for products relevant to Linux gamers and enthusiasts but also commonly reviewing servers/workstations and embedded Linux devices. Michael has written more than 10,000 articles covering the state of Linux hardware support, Linux performance, graphics hardware drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org automated testing software. He can be followed via and or contacted via .
Latest Linux Hardware Reviews
  1. CompuLab Intense-PC2: An Excellent, Fanless, Mini PC Powered By Intel's i7 Haswell
  2. From The Atom 330 To Haswell ULT: Intel Linux Performance Benchmarks
  3. AMD Radeon R9 285 Tonga Performance On Linux
  4. Apotop Wi-Copy
Latest Linux Articles
  1. AMD Moves Forward With Unified Linux Driver Strategy, New Kernel Driver
  2. MSI: Update Your BIOS From The Linux Desktop
  3. NVIDIA vs. AMD 2D Linux Drivers: Catalyst Is Getting Quite Good At 2D
  4. 15-Way GPU Comparison With Mesa 10.3 + Linux 3.17
Latest Linux News
  1. Phoronix Test Suite 5.4 M3 Is Another Hearty Update
  2. GParted 0.20 Improves Btrfs Support
  3. EXT4 In Linux 3.18 Has Clean-ups, Bug Fixes
  4. Emacs 24.4 Has Built-In Web Browser, Improved Multi-Monitor Support
  5. NVIDIA's NVPTX Support For GCC Is Close To Being Merged
  6. KDE's KWin On Wayland Begins Using Libinput
  7. Khronos Releases OpenVX 1.0 Specification
  8. Linux Kernel Working Towards GNU11/C11 Compatibility
  9. Ubuntu 15.04 Is Codenamed After A Monkey: Vivid Vervet
  10. Following GCC, Clang Looks To Default To C11
Latest Forum Discussions
  1. Updated and Optimized Ubuntu Free Graphics Drivers
  2. Users/Developers Threatening Fork Of Debian GNU/Linux
  3. HOPE: The Ease Of Python With The Speed Of C++
  4. Bye bye BSD, Hello Linux: A Sys Admin's Story
  5. NVIDIA Presents Its Driver Plans To Support Mir/Wayland & KMS On Linux
  6. AMD Is Restructuring Again, Losing 7% Of Employees
  7. Open-Source AMD Fusion E-350 Support Takes A Dive
  8. Upgrade to Kaveri, very slow VDPAU performance