1. Computers
  2. Display Drivers
  3. Graphics Cards
  4. Memory
  5. Motherboards
  6. Processors
  7. Software
  8. Storage
  9. Operating Systems


Facebook RSS Twitter Twitter Google Plus


Phoronix Test Suite

OpenBenchmarking.org

How We Are Using Btrfs To Find Regressions Incredibly Fast

Michael Larabel

Published on 30 June 2010
Written by Michael Larabel
Page 1 of 2 - 9 Comments

In previous articles I have hinted that at Phoronix we are working to take advantage of the Btrfs file-system within the Phoronix Test Suite and Phoromatic to provide an interesting feature that will further expand our automated testing capabilities, but how does this file-system come into play? Well, here is what's being worked on and it should be of terrific value to many people.

One of the features of Btrfs not found in other Linux file-systems (like EXT4) is support for copy-on-write snapshots / sub-volumes. With this Btrfs snapshotting support is the ability to mount different snapshots in a very easy manner and since they are copy-on-write, the disk storage requirements are not extreme. The Btrfs snapshot ability is being taken advantage of by Red Hat to provide Linux system rollback support whereby a Btrfs snapshot is created before a yum/RPM transaction takes place and if the installed package(s) wreak havoc on the system or something goes awry, you can simply choose an earlier snapshot to set your system's software back to an earlier state following a system reboot. We though are leveraging Btrfs snapshots in a different manner.

First off, for those not familiar with Phoromatic, it is a Phoronix Test Suite component that provides a remote management system and "allows the automatic scheduling of tests, remote installation of new tests, and the management of multiple test systems all through an intuitive, easy-to-use web interface. Tests can be scheduled to automatically run on a routine basis across multiple test systems. The test results are then available from this central, secure location." Part of Phoromatic is the Phoromatic Tracker, which uses Phoromatic to track the performance of a software component over the course of time in hopes of spotting any regressions and providing better analysis of how code changes are affecting the overall performance. Phoromatic / PTS / Phoromatic Tracker can also be used for functional and unit testing.

We have provided public implementations of Phoromatic Tracker for tracking the performance of the mainline Linux kernel and bleeding edge Ubuntu packages on a timed daily basis, but Phoromatic Tracker can also test software changes on a per-commit basis or when triggered by an external script. Our automated software testing stack has found performance regressions in the past with these trackers, but then we, the developers, or third-parties still need to go back and find the commit that caused the problem (for trackers running on a timed or non-per-commit basis), etc. Within the Phoronix Test Suite we already have a module for autonomously bisecting regressions by leveraging Git, which we have used already to locate regressions within the Linux kernel with next to no manual intervention, but by having Phoromatic/PTS play with Btrfs, we are working on something much greater.

Coming to Phoromatic will be the ability for the Phoronix Test Suite to automatically create a Btrfs snapshot on each of the test nodes whenever a test is to be run whether it be on a daily basis with some of the trackers or on a per-commit basis with some of the future public trackers and how Phoromatic is being used privately by some organizations. For example, with our Ubuntu Tracker this will leave us with Btrfs snapshots of the latest Ubuntu packages each day and for the Linux kernel this will leave us with the kernel from that respective day. Since the snapshots are copy-on-write, the disk requirements are not too big and thus quite a number of snapshots will be able to be kept around locally.

In other words, users will be able to access their tracked component(s) from any state in the past. Unlike using git-bisect and relying upon the package source and rebuilding any software and reinstalling it or even just reinstalling Deb/RPM packages, with this approach it's just a matter of remounting the Btrfs partition and in some cases a reboot. This makes it incredibly faster to do operations like automatically bisecting regressions in a binary search and more reliable as you are using the software in the same exact state that it was left previously when running tests on a given date/commit.

From the Phoromatic web-interface we will additionally be exposing a new set of options whereby when a regression is spotted in the tracked component, when there is "downtime" on a given test node (i.e. there are no scheduled tests at the moment) and a regression was recently spotted whether it be minutes or days ago, to be able to rollback the system temporarily to that spot to be able to run an expanded set of tests. For example, with the Linux kernel tracker we could be running just a few CPU tests as part of the scheduled test execution queue, but when a performance drop is spotted in one of the CPU tests, next time that test node is free it could rollback the system to the regressed spot and automatically run a greater number of CPU tests in hopes of being able to provide the developer with more information and numbers regarding the regression to better narrow down the scope of the problem or quantifying the repercussions of an intended code change. As another example, when a graphics regression is spotted, the Phoromatic-controlled systems could then go back and run more graphics tests or to run the same graphics test(s) with different resolutions and/or other test options. This would save on having to run a huge number of tests everyday or with every commit, but testing now becomes much smarter when a potential problem is detected.

But there's still more...

<< Previous Page
1
Latest Linux Hardware Reviews
  1. AMD Launches New FX CPUs, Cuts Prices On Existing Processors
  2. Preview: AMD's FX-9590 Eight-Core At Up To 5.0GHz On Linux
  3. Intel Launches The Core i7 5960X, Mighty Powerful Haswell-E CPUs
  4. AMD Radeon R9 290: Gallium3D vs. Catalyst Drivers
Latest Linux Articles
  1. LLVM Clang 3.5 Brings Some Compiler Performance Improvements
  2. Ondemand vs. Performance CPU Governing For AMD FX CPUs On Linux 3.17
  3. How Intel Graphics On Linux Compare To Open-Source AMD/NVIDIA Drivers
  4. The Fastest NVIDIA GPUs For Open-Source Nouveau With Steam Linux Gaming
Latest Linux News
  1. POCL Is Maturing Well For Running OpenCL On The CPU
  2. New Group Calls For Boycotting Systemd
  3. The Features To Find With The Imminent Release Of LLVM/Clang 3.5
  4. Borderlands 2 Is Coming To Linux
  5. The Witcher 2 Ups The Performance More & Works Around Catalyst Bug
  6. Running Gallium3D's LLVMpipe On The Eight-Core 5GHz CPU
  7. Trying Intel OpenCL On Linux For Video Encoding
  8. GSoC 2014 Yielded Some Improvements For Mesa/X.Org This Year
  9. webOS Lives On As LuneOS With New Release
  10. Marek Lands Radeon Gallium3D HyperZ Improvements
Latest Forum Discussions
  1. Lennart Poettering Talks Up His New Linux Vision That Involves Btrfs
  2. nv and xorg.conf under Debian PPC
  3. AMD graphics doesn't work with AMD Catalyst drivers
  4. Best Radeon for a Power Mac G5?
  5. The dangers of Linux kernel development
  6. Updated and Optimized Ubuntu Free Graphics Drivers
  7. AMD Releases UVD Video Decode Support For R600 GPUs
  8. SSD seems slow