Linux 2.6.23-rc2 Kernel Performance

[Michael]

Over the past three years at Phoronix we have established strict standards for Linux benchmarking that does involve multiple runs, managing processes, and we even reformat the hard drive(s) and do a complete reinstall of the Linux distribution and updates on each test system in between articles / rounds of testing.

I have commented more on our benchmarking process in other threads here on the forums and in some of our other articles. There is no single page that lists all the steps involved and all of the work that goes into our testing. Routinely we make minor tweaks and other refinements to our testing procedures, but as the executive editor I personally oversee each and every benchmark. We also maintain a variety of scripts for managing and automating the benchmarks while ensuring accuracy. In the near future I will work on a manifest that contains all of this information

[linuxfool]

Incomplete benchmarks

Phoronix: Linux 2.6.23-rc2 Kernel Performance

While the Linux 2.6.23 kernel is only weeks into development, it's already generated quite a bit of attention. ...
. With all of this activity surrounding the Linux 2.6.23 kernel we've decided to conduct a handful of benchmarks comparing the Linux 2.6.20, 2.6.21, 2.6.22, and 2.6.23 kernel releases so far.

http://www.phoronix.com/vr.php?view=10717

Is there a link missing from this article? I would expect it to explain the benchmarking approach, i.e. how many runs, what efforts are taken to eleminate or account for other processes distorting results, etc. etc.

I'd also expect to see the standard deviation for every result.

Without the standard deviation, or some measure of variance, and an explanation of methodology, it is difficult (polite way of saying practically impossible) to conclude anything.

If this is all 'mumbo-jumbo' then I'd recommend looking at "Statistics Hacks" by Bruce Frey, published by O"Reilly. It helped me a lot.

You aren't alone in not providing enough information, so please don't think I am trying to pick on you. If you feel like being beaten up for it, go have a look at Zed Shaw (http://www.zedshaw.com/rants/programmer_stats.html), but I would recommend "Statistics Hacks" as less painful and more productive :-)

LF

[user5124]

and we will have an article looking just at the scheduler in the future.

Great! Looking forward to it.

[hdas]

Indeed, I agree that it wasn't supposed to be a scheduler comparison and just about performance in single demanding applications. Aside, I believe its pretty difficult to compare schedulers as such. But then again, many people after reading the article may be tempted to conclude that there is not really much incentive to upgrade the kernel (and rightly so ), but I believe there is actually quite a marked difference (with the new schedulers of course ;-). I am sure in the real world one will find quite a few subtle but exciting improvements with the new kernels - many a things are not show stoppers, but definitely nagging sometimes (especially stuttering audio, or a drop in game fps ;-).

An off topic comment. This about dual core processors, smp and such. When I bought the computer (notebook with 1.6ghz centrino duo), I knew pretty much that I would be way better off with a single 3.2ghz or even a 2.8 or 2.6 ghz cpu, but I was excited at the breadth of things. Over the time, I have indeed been made to regret that (especially when I run some of my brute force computational programs). I somehow always managed to console myself with the thought that two is better than one. Few days back, for resolving my acpi issues, I had disabled my smp and to my surprise, I found that there is no performance change whatsoever, at least upto 2 or 3 moderately heavy tasks! There was absolutely no change in benchmarks (I atleast expected 1% ). Of course I could see the struggle when I started multiple heavy tasks and I could see it barely pulling inside games. But then again, when do I ever do such a harsh thing. Left me somewhat disappointed at my dual core decision ;-).

[buckyball]

Multitasking benchies?

I really appreciate the efforts of Phoronix to present the new and emerging technology of interest to the Linux community.

The only thing that would make me happier would be the use of meaningful multitasking / multithreading benchmarks in comparing things like kernel releases.

I believe that the kernel has evolved to the point that performance in single-threaded and/or compute-bound applications vary little (as this round of tests seem to demonstrate).

Take a box running LAMP software and beat on it some. This is where differences in things like process and I/O schedulers will really show up. Thanks!

[joshuapurcell]

I read the recent interview with Con Kolivas about why he left kernel development, but I didn't see any mention of CFS. I'm surprised by this since it seemed the whole reason for his departure related to the kernel scheduler. Maybe I just missed where the CFS name popped up in his interview.

I think Con Kolivas left for the wrong reasons, but it is a decision that he had to make for himself under the circumstances. I also make no claim to understand CFS and how it relates to the older schedulers. I wonder though how the inclusion of this new scheduler relates to Con Kolivas and his reasons for leaving... anyone have anything on this?

EDIT: Here is a link to the interview mentioned above:

http://apcmag.com/6735/interview_con_kolivas

[Michael]

The point of these benchmarks were to simply compare the performance of the Linux 2.6.23-rc2 kernel against some of the recent kernel releases in some benchmarks commonly used by Phoronix.

The article isn't intended to be just a CFS comparison (hence it's called "Linux 2.6.23-rc2 Kernel Performance" not "CFS Scheduler Performance" or "Linux Scheduler Comparison"). CFS is just one of the additions to the Linux 2.6.23 kernel and we will have an article looking just at the scheduler in the future.

[oneman]

The benchmarking in this article missed the point of the new scheduler so much I wanted to cry but all I could do was laugh.


What would really be interesting and usefully is 2.6.22-ck1 vs 2.6.23-X with CFS. Everyone knows both CFS and SD are better than the old stock. Even tho CFS has been chosen over SD, there is still much to be gained by comparing them. Specifically in context of gaming workloads.

[lenrek]

Maybe is just me, from the article, I don't really find it worth excited for...

[user5124]

Well, the benchmarks done by phoronix were good for showing what the CFS schedular does in regards to a non-loaded system, but ideally, as above, what would have been better would have been output performance tests on a system running multiple cpu-hog processes.


Also, I registered just to post about this, and say that Phoronix is great. Happy to have it in my "browse all the time" bookmarks.