Bringing Up Advancements In Compiler Testing
With Phoronix Test Suite 3.8-Bygland, which will be officially released this month, there are some last-minute changes being pushed for this quarterly release cycle. There's already a number of new features in Phoronix Test Suite 3.8, but being pushed in recent days has been the start of advancements to enhance the compiler testing abilities of the Phoronix Test Suite and its reporting of compiler options and other system settings.
One of the Phoronix Test Suite commercial customers is sponsoring the development of several new features for Phoronix Test Suite 3.8 and the major Phoronix Test Suite 4.0 release later in the year. One of their big areas of interest is with compiler-based testing, and fortunately they're kind enough to let the sponsored work be pushed directly into the open-source/community Phoronix Test Suite code-base. What's landed in the Git repository already for the forthcoming Phoronix Test Suite 3.8 is automated reporting of compiler options.
Through a compiler-masking method, compiler options are being seamlessly monitored and recorded by the Phoronix Test Suite upon each test installation. That information is then later automatically analyzed looking for compiler differences between test runs and to then pass the relevant information onto pts_Graph for rendering. Here's the brief compiler testing description from the Phoronix Test Suite documentation.
A majority of the test profiles provided by OpenBenchmarking.org to the Phoronix Test Suite are source-based tests. Relying upon the upstream source-code for each program under test allows for the tests to be easily brought to new platforms and architectures, avoids any out-of-tree / non-default packaging differences by different distributions and operating systems, and to allow the Phoronix Test Suite user to easily test new compilers and/or compiler options. For the source-based tests, the Phoronix Test Suite relies upon a compiler (e.g. GCC, LLVM/Clang, Sun Studio, Open64, et al) being present on the system under test. The Phoronix Test Suite does respect CC/CXX environment variables and test profiles are expected to honor CFLAGS/CXXFLAGS and other compiler settings.
As of Phoronix Test Suite 3.8, a compiler masking method is utilized for logging compiler options and other settings with each test profile installation. Prior to installing an open-source test, the Phoronix Test Suite determines the intended compiler to be used based upon the pre-set environment variables or the pre-set compiler(s) within the PATH environment variable. The Phoronix Test Suite then masks the compiler to ensure that any options/flags submitted to the compiler are first passed through pts-core so that they can be logged for later use, then is linked to the original, intended compiler. Additionally, other compiler binary names of the same type are blacklisted to prevent their un-intended use (i.e. if a test profile has hard-coded gcc in its build script, but clang is set as the compiler via CC, a sym-link will automatically be created from gcc to the masked clang for the duration of its test installation).
The logged compiler data is then used by the Phoronix Test Suite following the test execution process for automated result analysis. If there is a detected change in compiler settings, the differences are automatically reported to the test result graphs. Additionally, key compiler information (e.g. the compiler optimization level and key libraries that were linked at compile-time) is reported as a footnote on result graphs. The Phoronix Test Suite handles all of this in a fully automated manner.
Below are a few sample graphs just intended to highlight the new compiler reporting that's handled automatically by the Phoronix Test Suite (the results are just sample data locally).
From this first example, it was a simple run of just running the same test multiple times. As you can see, the new addition to 3.8-Bygland when running this test profile is that some of the compiler information is now auto-reported as a footnote. It shows that by default OpenMP is being used and the compiler optimization level (-O2). This is especially helpful for new users that may not be familiar with the specifics of a test profile. This is all done automatically without requiring any test profile updates or other changes. As has always been the case, the compiler version and other logs (e.g. cc -v) is already captured and available from the system information area.
With the second example of the same test profile, the Phoronix Test Suite determined that the compiler's optimization level was being changed between runs. As a result, on each of the result bars it now shows the varying information (it's also checked to verify that the result identifier doesn't contain this information, so it doesn't appear redundant). As a foot-note it still indicates that OpenMP is being used by the test, since that was not changed between test runs.
In this example, the Phoronix Test Suite detected that the compiler was being changed out (via CC) when the test was being re-installed.
These compiler reporting improvements will also be visible on the graphs generated by OpenBenchmarking.org in the coming weeks. The version of pts-core running OpenBenchmarking.org simply needs to synced again to benefit from this work. This latest work is found in the Git code-base right now and will be featured in Phoronix Test Suite 3.8-Bygland Milestone 3 to be released in the next week. The official 3.8 release is still expected by month's end to complete the quarter.
Stay tuned for more reporting advancements plus other features. Additionally, any questions or feedback by the community is welcome on how this work can be expanded or be made more beneficial.