How We Are Using Btrfs To Find Regressions Incredibly Fast

The other part of this work -- and perhaps more importantly -- is the collection of snapshots that will be around from the Phoromatic Trackers. In the long term we will provide a public interface whereby independent users can tap into these snapshots to take advantage of incredibly fast bisecting. As you have no code recompiling or other leg work to do at each step of the way as Phoromatic and the Phoronix Test Suite knows which snapshot correlates to what commit or date and how to handle all of this in an automatic manner, it will be significantly faster than bisecting at the code level. So you could find a bug in the Linux kernel (as an example) but rather than spending all of that time bisecting yourself or using our PTS bisect module where you need to wait around for the kernel build process to complete with each step, you could just create a Phoronix Test Suite test profile that illustrates your bug. You would just need to upload this custom test profile via the public interface for the given tracker along with the good/bad points, and next time there is a test node within that tracker is available, it would automatically do the bisecting for you using these Btrfs snapshots that it has created in the past. Assuming the bug can be demonstrated on the given system, Phoromatic will find it using the provided test profile and then show you where the regression was introduced. Afterwards, the test node can continue with bisecting other bugs and then roll forward the system when it is time for its new scheduled testing.

Another option being considered is expanding these capabilities beyond just Phoromatic Tracker. Within our test farm (or the test farms of those organizations using the Phoronix Test Suite commercial services) we could have a system that does nothing but run virtual machines with each instance doing nothing building and installing a popular upstream software component on a per-commit basis (i.e. GCC, the Linux kernel, X Server, DDX drivers, Mesa, etc) and having the Phoronix Test Suite trigger the Btrfs snapshot. When a user wishes to then bisect a regression with a custom test profile on one of the software components that's being tracked, the VM that's tracking the specific component with its Btrfs snapshots would then be copied automatically to an available test node on the network and the VM then executed and carried out just like mentioned above. While Btrfs is what's being talked about, the code will be abstracted enough where ZFS snapshots or potentially even QCOW2 snapshots for virtual machines could be used instead.

Yet another avenue being explored is for handling Phoromatic Trackers that are closely tied to multiple packages while being able to test the package set as a whole in a very efficient manner and still have the capabilities to narrow down any regressions. For example, with the modern Linux graphics stack consisting of the in-kernel DRM, libdrm, and Mesa all being developed at the same time, if there was a Phoromatic Tracker that pulled the latest Git code for each of these components on a daily basis and a regression then spotted in one of the tests, it could be difficult to isolate in what component the regression originated. With this Btrfs-leveraged model, snapshots could be created after installing the updated DRM, another after the libdrm, and the final one after Mesa was updated and is ready for testing. The scheduled testing would occur on only the last state, but if a performance or functional regression was found compared to the previous day, Phoromatic could automatically trigger the affected test node(s) to go back and run the impacted tests on the interim snapshots to quickly isolate in what component there is the problem.

Well, that is the basis (without divulging all of our future plans) of this new feature that is already being worked on internally and a prototype for public demonstration is in sight. As a one sentence takeaway, the Phoronix Test Suite will provide very fast bisecting support of regressions by leveraging Btrfs to perform bisecting of ready test states at the file-system level. This is one of several features being worked on for Phoronix Test Suite 3.0. While we reference the public trackers and using this feature on key open-source software packages within this article, Phoromatic can already be used privately by any organization for testing any software component and this will continue to be the case going forward. If you have any requests, feedback, or your organization would like to sponsor this development or deploy such capabilities on a corporate intranet, contact us or let us know in the forums.