The Past 12 Linux Kernels Benchmarked

[Tillin9]

I hate to sound ungrateful, I mean so far no one has done these sorts of benchmarks, and they're useful. However, its possible to greatly improve how useful these numbers could be with a little extra care. First, what number of tests did you run? Was it 1, 10, 100? If it is stated in the article I missed it, but the average of 100 tests carries a lot more weight than just 1. Ideally, an error bar, standard deviantion, or confidence interval would also be included. At least some measure of how sure you are that the result is indicative of the measurement. This is especially important in cases where the benchmarks are very close, since it lets us run a correlation test to make sure they are actually statistically different.

Also, on a more pedantic level is the number of significant figures in the graphs. It should reflect the precision of the score. The general rule of thumb is everything but the last digit should be fixed on repeat of measurement (or withing the standard deviation). A score of 28.1293 for example, means that every time you get 28.129x. Ssam's data for example would only warant a score of 63 MB/s, even though the actual average of the three scores is 63.47 MB/s since both the units digit varied between measurements and the standard deviation is roughly ~.75 MB/s.

Kcalc (the standard KDE calculator app) and a wide variety of Linux graphics packages can compute this stuff automatically.

[Michael]

I hate to sound ungrateful, I mean so far no one has done these sorts of benchmarks, and they're useful. However, its possible to greatly improve how useful these numbers could be with a little extra care. First, what number of tests did you run? Was it 1, 10, 100?

It's the average of four runs for each test.

[cruiseoveride]

At least we know Linux is not getting slower, like windows.