Google Has Been Quietly Developing "GFR" To Debug Vulkan GPU Hangs/Crashes

Last year Google quietly open-sourced the Graphics Flight Recorder (GFR) for sorting out GPU hangs and crashes. GFR is implemented as an implicit Vulkan layer that works on both Windows and Linux.

Google has been developing the Graphics Flight Recorder with a focus on Stadia cloud gaming use-cases for developers to sort out GPU hangs/crashes. This Vulkan layer adds completion tags to Vulkan command buffers so when an error is detected those command buffers can be dumped -- particularly around knowing where there was the last completed or first incomplete buffer.

This GFR Vulkan layer is low overhead enough to keep it activated until happening to run into a GPU hang/crash and will then automatically generate the log dump. The dump can then be fed back into one of its scripts to regenerate the source files.

One of the few downsides to the Graphics Flight Recorder though is that it depends upon the VK_AMD_buffer_marker Vulkan extension, which isn't widely supported at the moment -- but easy to implement for the open-source drivers. The Graphics Flight Recorder is hosted on GitHub.

GFR was released without much fanfare but Igalia developer Danylo Piliaiev spotted it. Danylo noted that GFR has already been useful for working out some hangs and encourages other developers to give it a shot.