Experimental Linux Patches Allow Kernel Tracing To Work Past Reboots/Crashes

Steven Rostedt this week posted some interesting albeit experimental patches for the Linux kernel to support persistent traces that work across a reboot or crash.

The tentative patches allow for mapping a ring buffer instance across a reboot or crash rather than losing that information that can be useful in debugging issues. Rostedt explained with the initial kernel patches:

"This is a way to map a ring buffer instance across reboots. The requirement is that you have a memory region that is not erased. I tested this on a Debian VM running on qemu on a Debian server, and even tested it on a baremetal box running Fedora. I was surprised that it worked on the baremetal box, but it does so surprisingly consistently.
...
Finally, this is still a proof of concept. How to create this memory mapping isn't decided yet. In this patch set I simply hacked into kexec crash code and hard coded an address that worked for one of my machines (for the other machine I had to play around to find another address). Perhaps we could add a kernel command line parameter that lets people decided, or an option where it could possibly look at the ACPI (for intel) tables to come up with an address on its own.

Anyway, I plan on using this for debugging, as it already is pretty featureful but there's much more that can be done.

Basically, all you need to do is:

echo 1 > /sys/kernel/tracing/instances/boot_mapped/events/enable

Do what ever you want and the system crashes (and boots to the same kernel). Then:

cat /sys/kernel/tracing/instances/boot_mapped/trace

and it will have the trace.

I'm sure there's still some gotchas here, which is why this is currently still just a POC."

See this patch series for this interesting work that will hopefully evolve and at some point make it into the mainline Linux kernel.