Since Friday there's been a number of Phoronix articles about a very bad power regression in the mainline Linux kernel, which is widespread, Ubuntu 11.04 is one of the affected distributions, and has been deemed a bug of high importance. This yet-to-be-resolved issue is affected Linux 2.6.38 and 2.6.39 kernels and for many desktop and notebook systems is causing a 10~30% increase in power consumption. Nevertheless, this is not the only major outstanding power regression in the mainline tree, there is another dramatic regression now spotted as well that is yet-to-be-fixed.
Since the discovery last week of the Linux 2.6.38 and 2.6.39 kernels going through excessive amounts of power compared to 2.6.37 and earlier, each day and practically all day since that time has been working on Linux power consumption tests. Power consumption benchmarks are not normally an area we focus on nor do many others, but since the inadvertent discovery of it when testing out the power consumption of past Ubuntu Linux releases, a lot of time has spent investigating the matter within the kernel. In order to do such, there's been continued improvements to the Phoronix Test Suite, Phoromatic, OpenBenchmarking.org, and the PTS Commercial scripts for better enhancing the power testing, more improvements to multi-point automated regression bisecting, etc. The Phoronix Test Suite has already been able to monitor and log the power consumption (along with temperatures, fan speeds, I/O wait, system load, etc) for any test profile/suite being run by using the system monitor module, but now there is more. Thanks to working on that Easter weekend, coming to fruition because of that today is the discovery of another regression while still working on finding the first commit causing a power regression.
This regression for at least a ThinkPad notebook is even more serious than the first discovery and worst of all it has been living in the mainline tree for nearly one year. This second regression was introduced during the development of the Linux 2.6.35 kernel back in the summer of 2010. For some good news, this second regression may be harder to hit. In last week's Ubuntu power consumption tests, there isn't a noticeable regression in Ubuntu 10.10 (the first release using Linux 2.6.35) for all tests and all systems, but in some there tests when better dissecting between the mess of colors, there is some change for some environments.
How this second regression came about was when using Ubuntu 8.04.4 LTS on the ThinkPad R52 and then automatically benchmarking every kernel release from Linux 2.6.24 to present. This is with continuing a focus on power measurements. With Linux 2.6.24 having been released in January of 2008, this provides over three years of Linux kernel power performance numbers for stable releases. For the sake of testing and reproducibility by external parties, the convenient Ubuntu mainline Linux kernel repository was used for the mainline i686 kernel Debian packages.
Unlike the original Ubuntu Linux power consumption tests, this isolates any user-space differences impacting the power consumption as well as other differences like the Ubuntu switch from the EXT3 to EXT4 file-systems and going from a user-space mode-setting driver to kernel mode-setting for graphics, etc.
With this expanded round of power testing, the Linux 2.6.37 to Linux 2.6.38 regression is still shown, but it also uncovered a very noticeable differentiation in power consumption between the Linux 2.6.34 and 2.6.35 kernels too. Under idle on this test system, it equates to a 20% difference in power consumption and then the 2.6.37/2.6.38 regression tacks on another 6% in this particular test profile.
The graphs are on the following pages, but all of them illustrate the clear trend: The pre-2.6.35 kernels are all running at right around the same power level within a reasonable milliwatt range of each other on the various tests. [Thus also showing the tests and the Phoronix Test Suite aren't being spontaneous with its results or anything along those lines.] However, with the Linux 2.6.35 kernel, the power consumption goes up noticeably. That first power regression remains in Linux 2.6.36 and 2.6.37, and now with 2.6.38 the power consumption goes up even more. As our tests have shown, the regression is still outstanding with the current Linux 2.6.39 Git code that is in the middle of its development cycle.
The individual commits causing each regression are still being tracked down, but the 2.6.38 regression affecting so many hardware/software configurations still looks like it might possibly be within the memory management sub-system (see this morning's news posting for additional details). It is also possible that the two regressions are linked, but so far, that does not appear to be the case.
On the following pages are the results being shown that illustrate the 2.6.38 and 2.6.35 power consumption regressions. There's also the results of the other kernels going back to Linux 2.6.24, but some of the older kernels in between were left out since their results were the same as the 2.6.24-to-2.6.34 kernels and were just leading to cluttered graphs. These graphs are also rendered using the new color allocator in bilde_renderer for Phoronix Test Suite 3.2-Grimstad to hopefully better differentiate some of the results. If there are any problems, all of the graphing code is open-source via the Phoronix Test Suite and patches are happily accepted upstream. You can also pull down all of these test results from OpenBenchmarking.org via the 1104256-GR-RADEONPOW94 result file if you wish to manipulate the graphs or just to better analyze the massive data-set.