Bringing Up Advancements In Compiler Testing
For those interested in compiler testing and tuning performance, there's some notable advancements to share that will benefit the community.
With Phoronix Test Suite 3.8-Bygland, which will be officially released this month, there are some last-minute changes being pushed for this quarterly release cycle. There's already a number of new features in Phoronix Test Suite 3.8, but being pushed in recent days has been the start of advancements to enhance the compiler testing abilities of the Phoronix Test Suite and its reporting of compiler options and other system settings.
One of the Phoronix Test Suite commercial customers is sponsoring the development of several new features for Phoronix Test Suite 3.8 and the major Phoronix Test Suite 4.0 release later in the year. One of their big areas of interest is with compiler-based testing, and fortunately they're kind enough to let the sponsored work be pushed directly into the open-source/community Phoronix Test Suite code-base. What's landed in the Git repository already for the forthcoming Phoronix Test Suite 3.8 is automated reporting of compiler options.
Through a compiler-masking method, compiler options are being seamlessly monitored and recorded by the Phoronix Test Suite upon each test installation. That information is then later automatically analyzed looking for compiler differences between test runs and to then pass the relevant information onto pts_Graph for rendering. Here's the brief compiler testing description from the Phoronix Test Suite documentation.
A majority of the test profiles provided by OpenBenchmarking.org to the Phoronix Test Suite are source-based tests. Relying upon the upstream source-code for each program under test allows for the tests to be easily brought to new platforms and architectures, avoids any out-of-tree / non-default packaging differences by different distributions and operating systems, and to allow the Phoronix Test Suite user to easily test new compilers and/or compiler options. For the source-based tests, the Phoronix Test Suite relies upon a compiler (e.g. GCC, LLVM/Clang, Sun Studio, Open64, et al) being present on the system under test. The Phoronix Test Suite does respect CC/CXX environment variables and test profiles are expected to honor CFLAGS/CXXFLAGS and other compiler settings.
Below are a few sample graphs just intended to highlight the new compiler reporting that's handled automatically by the Phoronix Test Suite (the results are just sample data locally).
From this first example, it was a simple run of just running the same test multiple times. As you can see, the new addition to 3.8-Bygland when running this test profile is that some of the compiler information is now auto-reported as a footnote. It shows that by default OpenMP is being used and the compiler optimization level (-O2). This is especially helpful for new users that may not be familiar with the specifics of a test profile. This is all done automatically without requiring any test profile updates or other changes. As has always been the case, the compiler version and other logs (e.g. cc -v) is already captured and available from the system information area.
With the second example of the same test profile, the Phoronix Test Suite determined that the compiler's optimization level was being changed between runs. As a result, on each of the result bars it now shows the varying information (it's also checked to verify that the result identifier doesn't contain this information, so it doesn't appear redundant). As a foot-note it still indicates that OpenMP is being used by the test, since that was not changed between test runs.
In this example, the Phoronix Test Suite detected that the compiler was being changed out (via CC) when the test was being re-installed.
These compiler reporting improvements will also be visible on the graphs generated by OpenBenchmarking.org in the coming weeks. The version of pts-core running OpenBenchmarking.org simply needs to synced again to benefit from this work. This latest work is found in the Git code-base right now and will be featured in Phoronix Test Suite 3.8-Bygland Milestone 3 to be released in the next week. The official 3.8 release is still expected by month's end to complete the quarter.
Stay tuned for more reporting advancements plus other features. Additionally, any questions or feedback by the community is welcome on how this work can be expanded or be made more beneficial.
Latest Linux News
Latest Articles & Reviews
Most Viewed News This Week