1. Computers
  2. Display Drivers
  3. Graphics Cards
  4. Memory
  5. Motherboards
  6. Processors
  7. Software
  8. Storage
  9. Operating Systems


Facebook RSS Twitter Twitter Google Plus


Phoronix Test Suite

OpenBenchmarking.org

Autonomously Finding Performance Regressions In The Linux Kernel

Michael Larabel

Published on 22 October 2009
Written by Michael Larabel
Page 2 of 2 - 44 Comments

This bisect module starts by analyzing the result file and comparing the selected test(s) that illustrate a significant performance drop (or gain). Once that has been done and the good/bad revisions tested on the running test system, git-bisect is automatically called and handled by this Phoronix Test Suite module. With each bisect, the test with any specific test arguments is called and this module then determines whether that revision resulted in a "good" or "bad" run. The bisect module continues until the commit causing the problem is found. Now and into the future we will also work to expand these capabilities and provide more features during the bisecting process.

While this module was written to locate a Linux kernel performance regression, it can be used with any Git-based package that is suitable for testing by the Phoronix Test Suite (read: we can just as easily track down performance regressions in the X Server, Mesa, and drivers). Of course, an external build/install script needs to be written for setting up the environment, but this module should be very extensible. In Phoronix Test Suite 2.2 and future releases, the capabilities of the bisect module and our analytical options within this GPLv3 code should expand. Of course, this is for finding a performance regression from the past. For properly monitoring the performance of system code as its developed, one could also run Phoromatic, which will have the capability of automatically calling test runs after a Git commit is made and to then report the results back to the central repository in real-time rather than debugging regressions well after the fact.

Did the Phoronix Test Suite end up being successful in tracking down this PostgreSQL performance regression in the Linux kernel? Oh yes it did. The very significant drop in PostgreSQL's performance in the Linux 2.6.32 kernel with default options can be attributed to this lone Git commit that is for a fix to address cache flushing in ext4_sync_file for the EXT4 file-system. This commit improves data integrity in the event of a power loss or other problem, but carries a high disk performance penalty. After the Phoronix Test Suite module reported that this was the faulty commit, it was manually confirmed too. When asked about the commit, Red Hat's Eric Sandeen shared:

"Hey, thanks for doing the digging :)

It is required for safe behavior with volatile write caches on drives.

You could mount with -o nobarrier and it would go away, but a sequence like write->fsync->lose power->reboot may well find your file without the data that you synced, if the drive had write caches enabled.

If you know you have no write cache, or that it is safely battery backed, then you can mount with -o nobarrier, and not incur this penalty.

-Eric"

Many kernels were built in the process of tracking down this PostgreSQL performance drop triggered by this default EXT4 change, but fortunately through the Phoronix Test Suite it required very little user intervention once the bisect module was running thanks to a combination of PTS Bardu and Git. Rather than having to use git-bisect manually and building out the software each time, the Phoronix Test Suite can cover it and just tell you if/where a performance regression occurs all while you are sitting back and watching. Look for this capability in Phoronix Test Suite 2.2 "Bardu" along with major GTK user-interface improvements, verification of statistical significance in benchmark results, and many other new features. You can expect to read more on these automated revision-traversing testing capabilities at Phoronix as its features are brought forward and extended to drive a new level of benchmarking in open-source software.

About The Author
Michael Larabel is the principal author of Phoronix.com and founded the web-site in 2004 with a focus on enriching the Linux hardware experience and being the largest web-site devoted to Linux hardware reviews, particularly for products relevant to Linux gamers and enthusiasts but also commonly reviewing servers/workstations and embedded Linux devices. Michael has written more than 10,000 articles covering the state of Linux hardware support, Linux performance, graphics hardware drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org automated testing software. He can be followed via and or contacted via .
Latest Linux Hardware Reviews
  1. Trying The Configurable 45 Watt TDP With AMD's A10-7800 / A6-7400K
  2. Sumo's Omni Gets Reloaded
  3. AMD A10-7800 & A6-7400K APUs Run Great On Linux
  4. Radeon Gallium3D Is Running Increasingly Well Against AMD's Catalyst Driver
Latest Linux Articles
  1. Intel's Latest Linux Graphics Code Competes Against OS X 10.9
  2. Intel Sandy Bridge Gets A Surprise Boost From Linux 3.17
  3. Open-Source Radeon Graphics Have Some Improvements On Linux 3.17
  4. CPUFreq Scaling Tests With AMD's Kaveri On Linux 3.16
Latest Linux News
  1. GNOME 3.14 Beta Released
  2. Mesa 10.3 Branched & RC1 Released, Mesa 10.4 On Master
  3. Intel Sandy Bridge Gains On Linux 3.17 Extend Beyond Graphics
  4. LinuxCon: What's Going On With Fedora.Next
  5. Canonical Joined The Khronos Group To Help Mir/Wayland Drivers
  6. EFL 1.11 Is A Big Milestone For Enlightenment Users
  7. DirectFB Updates GTK3 Support, Working Towards DirectFB 1.8
  8. Userptr Support Set For AMD Radeon GPUs In Linux 3.18
  9. NVIDIA Releases CUDA 6.5 As A Huge Update
  10. GNOME 3.14 Beta Makes GLSL Optional, Supports Wayland Gesture/Touch Events
Latest Forum Discussions
  1. Systemd 216 Piles On More Features, Aims For New User-Space VT
  2. Updated and Optimized Ubuntu Free Graphics Drivers
  3. AMD Offers Mantle For OpenGL-Next, Pushes Mantle To Workstations
  4. ATI CrossFire Does Not Support On This Platform When Enabling (Ubuntu Lucid)
  5. Dead Island for Linux (?)
  6. The dangers of Linux kernel development
  7. Remote gui not accessible in Phoronix Test Suite 5.2
  8. Next-Gen OpenGL To Be Announced Next Month