How-To: Find The Mesa Performance Regressions
Here's one of the ways that Phoronix finds the performance regressions within the Mesa / Gallium3D drivers in a very easy way.
As shown last week with The Grinch That Stole The Radeon Gallium3D Performance, it's not too difficult to find regressions within the open-source Mesa / Gallium3D drivers. Well, it's not hard to find regressions in many areas of the Linux stack, but in particular for Mesa/Gallium3D as even though this is a critical piece to the Linux / free desktop desktop, they don't currently rely upon any continuous integration infrastructure or any continued performance monitoring aside from the developers mostly pushing Piglit for looking towards functional problems. Of course, I've been working on my own means to address some of these problems, the Phoronix Test Suite stack can already:
- Leverage Phoromatic (soon a new version integrated atop OpenBenchmarking.org) to continually test Mesa drivers for OpenGL performance regressions, image quality comparisons, and other functional tests. This has been demonstrated with monitoring the daily performance of the Linux kernel and various other open-source projects for several years already. The Phoronix Test Suite can also log the GPU fence counts, GPU temperatures, GPU fan speeds, CPU usage, and various other system vitals all in a fully automated manner while tests are being executed. Phoromatic could issue Mesa tests on a per-commit basis to the Git repository or on a timed (daily) basis.
- Utilize the Phoronix Test Suite's bisecting capabilities that hook into git-bisect for automatically finding regressions by looking for changes in performance from a test profile and triggering git bisect automatically. This is how last week's Radeon performance regressions were tracked, finding Linux kernel bugs, and other Linux regressions.
- A combination of solutions encompassing the Phoronix Test Suite, as is done by Wine / CodeWeavers.
- Using MATISK. (What this article is talking about.)
MATISK is a Phoronix Test Suite module I demonstrated earlier this year for benchmarks thousands of revisions of a software project (it happened to be Mesa, again, in that article). Right now I'm in the process of benchmarking (well, temporarily delayed while in Vienna this week and waiting for the January advertising campaigns to kick in before publishing data) all the revisions of Mesa since 7.11 to see what other regressions and issues are outstanding for the upcoming Mesa 7.12/8.0 release. Phoronix Test Suite with the MATISK module can do this extremely easy as it's just fed a simple INI-based file with defining a custom context, what to do with each context, etc.
I've recently made some improvements to the MATISK module too that are now in the public Git repository in preparation for Phoronix Test Suite 3.8-Bygland in Q1'2012 (so if you want to try this out, use the Phorogit code). Below is a basic MATISK example (you can also auto-generate a template using phoronix-test-suite matisk.template) that's similar to what I'm using in testing the R600 Gallium3D driver.
; Sample INI Configuration Template For Phoronix Test Suite MATISK
So what it's doing is defining various parameters for the Phoronix Test Suite (result identifier, file name, test installation settings, etc) that is similar to what's set by Phoromatic or when running the Phoronix Test Suite in its batch mode. And then also in this INI file are various parameters that are used by the MATISK module for gathering contexts. A context could be virtually anything from a Git hash, a ISO date, a software (RPM/DEB) package name, a build identifier / UUID, or any other unique identifier. It could also be an array of various kernel module parameters, file-system mount options, or really any other string. The MATISK module then calls upon the Phoronix Test Suite to execute each of these after running the phoronix-test-suite matisk.run [the-INI-file] command. At one of the hook stages, you can then call upon an external executable file or a series of commands to then do something while receiving one of the unique contexts. Right now MATISK isn't focused upon having the MATISK files be widely distributed across platforms or for setting up initial environments from scratch, but based upon the direction of OpenBenchmarking.org and other Phoronix Test Suite, you can likely see the future direction and my grand plans.
In the case of this example, the contexts is a list of Git hashes between the 7.11 tag and master for the mainline Mesa git tree that's automatically being parsed from git log. Prior to running the test queue, one of the contexts (a Git hash) is being passed to a series of commands that pulls that specific Git revision, runs the Mesa build script, and installs the new drivers locally (FYI, this INI file example is intentionally quite basic and not fully complete). Then the tests are executed and the process repeated -- all in a fully automated manner. MATISK also attempts to automatically recover itself and the Phoronix Test Suite if one of the commands or scripts would like to reboot the system, such as for trying a different kernel or other low-level configuration change.
As with all of the Phoronix Test Suite related products, it's meant to be turnkey and fully automated once configured for any process. MATISK also has support for skipping a particular context based upon a script's exit code and other various configurable parameters. Feedback, as always, is welcome.
The current MATISK module in the Phoronix Test Suite is compatible with Linux, BSD, Mac OS X, and Solaris operating systems.
Any questions can be directed to the forums. Phoronix Media is also happen to engage with leading open-source projects gratis and does offer a variety of enterprise products and services for commercial organizations interested in continuous integration, performance monitoring, system analytics, and more.
More exciting Phoronix Test Suite / OpenBenchmarking.org features are also coming forward in 2012!
[ P.S. speaking of open-source graphics driver performance: it looks like there may be a Nouveau DRM performance change concerning ctxprogs. See this revised mailing list thread for more details. ]
Latest Linux Hardware Reviews
Latest Linux Articles
Latest Linux News
Latest Forum Discussions