Finding Bottlenecks In Your Linux Game, Application

If you are curious how Valve and Intel improved the OpenGL performance of their Mesa driver and also at the same time finding areas for performance optimizations within the Source Engine, here's some of the tools used.

The tools that seem to be predominantly used, based upon what Intel's Ian Romanick wrote in a blog comment and from what I've learned in email communications and while at Valve, are Sysprof and APITrace.

Sysprof isn't limited to just games or OpenGL workloads but is a system-wide performance profiler for Linux. Sysprof traces user-space and kernel-space calls, doesn't require the rebuilding of any software, supports saving/loading profiles, provides a straightforward user-interface, and does have command-line options. Intel and Valve developers used Sysprof to determine where the most amount of time within the system was being consumed and from there it's much easier to analyze the chunks of code keeping your prrogram the most easy.

Sysprof itself isn't a new tool and has been around for some years. Back in 2010 I wrote about the Karsk module for the Phoronix Test Suite that pairs automated benchmarking with Sysprof to automatically analyze different configurations, although the PTS Karsk module isn't publicly available at the moment.

When it comes to analyzing OpenGL usage and the different GL calls a program makes, the excellent utility for that is the open-source APITrace. APITrace is largely developed by the Intel Mesa developers as well as VMware (former Tungsten Graphics developers) and others. APITrace makes it easy to trace all of the calls made by a user-space program not only for OpenGL but also OpenGL ES, Direct3D, and DirectDraw. After a trace has been stored, re-traces can happen, OpenGL states can be inspected at anytime, traces can be visualized, and there's a handy Qt-based GUI as well. APITrace allows for trimming traces to only preserve the specific set of calls that illustrates a regression or other issues, compressing traces, and many other useful features.

Aside from supporting multiple 3D/drawing APIs, APITrace works not only under Linux but also Windows, Mac OS X, and Android. APITrace is very useful open-source tool and I'm a big fan of it as well.

In addition to APITrace, for OpenGL debugging there is GLIntercept, OpenGL's BuGLe, Tracy, and the closed-source AMD gDEBugger.