A majority of the test profiles provided by OpenBenchmarking.org to the Phoronix Test Suite are source-based tests. Relying upon the upstream source-code for each program under test allows for the tests to be easily brought to new platforms and architectures, avoids any out-of-tree / non-default packaging differences by different distributions and operating systems, and to allow the Phoronix Test Suite user to easily test new compilers and/or compiler options. For the source-based tests, the Phoronix Test Suite relies upon a compiler (e.g. GCC, LLVM/Clang, Sun Studio, Open64, et al) being present on the system under test. The Phoronix Test Suite does respect CC/CXX environment variables and test profiles are expected to honor CFLAGS/CXXFLAGS and other compiler settings.
As of Phoronix Test Suite 3.8, a compiler masking method is utilized for logging compiler options and other settings with each test profile installation. Prior to installing an open-source test, the Phoronix Test Suite determines the intended compiler to be used based upon the pre-set environment variables or the pre-set compiler(s) within the PATH environment variable. The Phoronix Test Suite then masks the compiler to ensure that any options/flags submitted to the compiler are first passed through pts-core so that they can be logged for later use, then is linked to the original, intended compiler. Additionally, other compiler binary names of the same type are blacklisted to prevent their un-intended use (i.e. if a test profile has hard-coded gcc in its build script, but clang is set as the compiler via CC, a sym-link will automatically be created from gcc to the masked clang for the duration of its test installation).
The logged compiler data is then used by the Phoronix Test Suite following the test execution process for automated result analysis. If there is a detected change in compiler settings, the differences are automatically reported to the test result graphs. Additionally, key compiler information (e.g. the compiler optimization level and key libraries that were linked at compile-time) is reported as a footnote on result graphs. The Phoronix Test Suite handles all of this in a fully automated manner.
From this first example, it was a simple run of just running the same test multiple times. As you can see, the new addition to 3.8-Bygland when running this test profile is that some of the compiler information is now auto-reported as a footnote. It shows that by default OpenMP is being used and the compiler optimization level (-O2). This is especially helpful for new users that may not be familiar with the specifics of a test profile. This is all done automatically without requiring any test profile updates or other changes. As has always been the case, the compiler version and other logs (e.g. cc -v) is already captured and available from the system information area.
With the second example of the same test profile, the Phoronix Test Suite determined that the compiler's optimization level was being changed between runs. As a result, on each of the result bars it now shows the varying information (it's also checked to verify that the result identifier doesn't contain this information, so it doesn't appear redundant). As a foot-note it still indicates that OpenMP is being used by the test, since that was not changed between test runs.
In this example, the Phoronix Test Suite detected that the compiler was being changed out (via CC) when the test was being re-installed.