1. Computers
  2. Display Drivers
  3. Graphics Cards
  4. Memory
  5. Motherboards
  6. Processors
  7. Software
  8. Storage
  9. Operating Systems


Facebook RSS Twitter Twitter Google Plus


Phoronix Test Suite

OpenBenchmarking.org

Autonomously Finding Performance Regressions In The Linux Kernel

Michael Larabel

Published on 22 October 2009
Written by Michael Larabel
Page 2 of 2 - 44 Comments

This bisect module starts by analyzing the result file and comparing the selected test(s) that illustrate a significant performance drop (or gain). Once that has been done and the good/bad revisions tested on the running test system, git-bisect is automatically called and handled by this Phoronix Test Suite module. With each bisect, the test with any specific test arguments is called and this module then determines whether that revision resulted in a "good" or "bad" run. The bisect module continues until the commit causing the problem is found. Now and into the future we will also work to expand these capabilities and provide more features during the bisecting process.

While this module was written to locate a Linux kernel performance regression, it can be used with any Git-based package that is suitable for testing by the Phoronix Test Suite (read: we can just as easily track down performance regressions in the X Server, Mesa, and drivers). Of course, an external build/install script needs to be written for setting up the environment, but this module should be very extensible. In Phoronix Test Suite 2.2 and future releases, the capabilities of the bisect module and our analytical options within this GPLv3 code should expand. Of course, this is for finding a performance regression from the past. For properly monitoring the performance of system code as its developed, one could also run Phoromatic, which will have the capability of automatically calling test runs after a Git commit is made and to then report the results back to the central repository in real-time rather than debugging regressions well after the fact.

Did the Phoronix Test Suite end up being successful in tracking down this PostgreSQL performance regression in the Linux kernel? Oh yes it did. The very significant drop in PostgreSQL's performance in the Linux 2.6.32 kernel with default options can be attributed to this lone Git commit that is for a fix to address cache flushing in ext4_sync_file for the EXT4 file-system. This commit improves data integrity in the event of a power loss or other problem, but carries a high disk performance penalty. After the Phoronix Test Suite module reported that this was the faulty commit, it was manually confirmed too. When asked about the commit, Red Hat's Eric Sandeen shared:

"Hey, thanks for doing the digging :)

It is required for safe behavior with volatile write caches on drives.

You could mount with -o nobarrier and it would go away, but a sequence like write->fsync->lose power->reboot may well find your file without the data that you synced, if the drive had write caches enabled.

If you know you have no write cache, or that it is safely battery backed, then you can mount with -o nobarrier, and not incur this penalty.

-Eric"

Many kernels were built in the process of tracking down this PostgreSQL performance drop triggered by this default EXT4 change, but fortunately through the Phoronix Test Suite it required very little user intervention once the bisect module was running thanks to a combination of PTS Bardu and Git. Rather than having to use git-bisect manually and building out the software each time, the Phoronix Test Suite can cover it and just tell you if/where a performance regression occurs all while you are sitting back and watching. Look for this capability in Phoronix Test Suite 2.2 "Bardu" along with major GTK user-interface improvements, verification of statistical significance in benchmark results, and many other new features. You can expect to read more on these automated revision-traversing testing capabilities at Phoronix as its features are brought forward and extended to drive a new level of benchmarking in open-source software.

About The Author
Michael Larabel is the principal author of Phoronix.com and founded the web-site in 2004 with a focus on enriching the Linux hardware experience and being the largest web-site devoted to Linux hardware reviews, particularly for products relevant to Linux gamers and enthusiasts but also commonly reviewing servers/workstations and embedded Linux devices. Michael has written more than 10,000 articles covering the state of Linux hardware support, Linux performance, graphics hardware drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org automated testing software. He can be followed via and or contacted via .
Latest Linux Hardware Reviews
  1. CompuLab Intense-PC2: An Excellent, Fanless, Mini PC Powered By Intel's i7 Haswell
  2. From The Atom 330 To Haswell ULT: Intel Linux Performance Benchmarks
  3. AMD Radeon R9 285 Tonga Performance On Linux
  4. Apotop Wi-Copy
Latest Linux Articles
  1. AMD Moves Forward With Unified Linux Driver Strategy, New Kernel Driver
  2. MSI: Update Your BIOS From The Linux Desktop
  3. NVIDIA vs. AMD 2D Linux Drivers: Catalyst Is Getting Quite Good At 2D
  4. 15-Way GPU Comparison With Mesa 10.3 + Linux 3.17
Latest Linux News
  1. Following GCC, Clang Looks To Default To C11
  2. Users/Developers Threatening Fork Of Debian GNU/Linux
  3. Linux 3.18-rc1 Released One Week Early With Many Changes
  4. The VC4 Gallium3D Driver Is Still Moving Along For The Raspberry Pi
  5. Direct3D 9 Support Might Land Within Mainline Mesa 3D Drivers
  6. OpenGL Preview Benchmarks For NVIDIA's GeForce GTX 970
  7. HOPE: The Ease Of Python With The Speed Of C++
  8. Vitesse: Using LLVM To Speed Up Databases
  9. AMD Is Restructuring Again, Losing 7% Of Employees
  10. Linux Testing Of The NVIDIA GeForce GTX 970
Latest Forum Discussions
  1. NVIDIA Presents Its Driver Plans To Support Mir/Wayland & KMS On Linux
  2. AMD Is Restructuring Again, Losing 7% Of Employees
  3. Bye bye BSD, Hello Linux: A Sys Admin's Story
  4. Open-Source AMD Fusion E-350 Support Takes A Dive
  5. Upgrade to Kaveri, very slow VDPAU performance
  6. ChromeOS Drops Support For EXT2/EXT3/EXT4 File-Systems
  7. Lennart Poettering On The Open-Source Community: A Sick Place To Be In
  8. The Slides Announcing The New "AMDGPU" Kernel Driver