MATISK: Benchmarking 1,000+ Revisions Of Mesa
With Phoronix Test Suite 3.4-Lillesand and the new MATISK support, over one thousand revisions of the Mesa graphics library were benchmarked. Two small performance optimizations were also noted.
MATISK is a new module to the 3.4-Lillesand development efforts and it is meant as the new code-path for driving context-based benchmarking in an automated manner. MATISK will unify core parts of the regression-detection, auto-bisection, and "Ekofisk" / Phoromatic components. MATISK will be controllable locally on a standalone system or when connected to a Phoromatic server instance, whether it be Phoromatic.com or the upcoming OpenBenchmarking.org-based version. MATISK is the current working-name for this module with the name coming from the word "Automatisk" (Norwegian for meaning automatic, following the other Phoronix Test Suite codenames of being Norwegian-derived words) and perhaps standing for "My Automated Test Infrastructure Setup Kit."
As input, MATISK accepts an INI-based file (there will also be a stable external API for those interested developers) that stores test parameters for automated execution and deals with context handling. Among the accepted parameters is the saved file name, the suite to execute, test description(s), whether to upload results to Openbenchmarking.org, etc. They are the basic test parameters that are commonly dealt with from the Phoronix Test Suite client, when setting up the batch testing mode, can be passed to the client via environmental variables, or also dealt with remotely from Phoromatic.
What MATISK deals better with locally now is support for context-based benchmarking. From the MATISK INI file, an array of contexts can be set (or an external file can be referenced for passing a list of contexts). What is a context in this sense? It can be practically any string. Some simple examples would be lists of Git commit hashes, various kernel command-line parameters to pass, or compiler build switches.
The context itself is really quite irrelevant to the Phoronix Test Suite client and MATISK module since, when supplying an array of contexts, the user can also set external scripts to run that are executed at pre-install, pre-run, post-install, and/or post-run times. The context, when about to be benchmarked, is then passed to the each of these external scripts. These scripts are then responsible for setting the appropriate context of the system under test. This allows for near-universal possibilities for automated testing.
With this Mesa testing, the MATISK INI loader referenced an external file of over 1,000 recent code commits (Git SHA1 hashes) to Mesa. A pre-install script was then written to pull the respective Mesa Git commit, build it, and then install. If the build failed, however, the appropriate exit code was submitted that tells the MATISK module to skip running benchmarks on that context. Additionally, since MATISK and the Phoronix Test Suite support automatic recovery upon system starts/reboots, after installing Mesa, the system was rebooted each time to ensure its GPU was in a clean state. After that, the Phoronix Test Suite runs as normal in a fully automated mode until the tests are complete, and then it moves onto the next context until the list has been exhausted. Bisecting of a context list is also being added so that the Phoronix Test Suite no longer has to depend upon external methods for regression bisecting as the Phoronix Test Suite does currently with using git-bisect.