Fedora 10 vs. Ubuntu 8.10 Benchmarks

[ian.woodstock]

Consistency in testing?

I was looking through the recent set of tests (vs Mac, vs Fedora, vs OpenSolaris, etc) I'm surprised to see that there doesn't seem to be a consistent set of test results published. While you seem to use the same test suite the results that are shown are cherry picked.
Perhaps this is just the highlights that shows the interesting comparisons, but it would be good to publish links to entire set or results, otherwise there's the chance that an unfair comparison is being mode - showing only the favorable results.

[Yfrwlf]

OS vs OS comparisons should really be done utilizing the distro's packages anyhows as that is how most people run those packages. In a hardware comparison test then the same version of OS should be used and the packages recompiled.

OS vs. OS can use the defaults, sure, but comparing performance differences between library versions is even better so that users will know that they can get better performance by installing the newer or older libraries.

Compilation doesn't have to be required, that doesn't really have anything to do with this, that's what binary packages are for, but whatever. The point is, you should be able to pinpoint the cause of slowdowns to differences in the libraries hopefully, but if those are the same then you know the performance loss lies elsewhere obviously.

You're probably right though, most users probably don't care so much, though maybe if it was easier to install newer libraries and compilation was required less of the time, more users would, and IMO that's where the focus should be is on the actual programs that cause the differences in performance. If you don't direct the problems to where they actually are then they'll never get solved.

[dm0rais]

Very nice, aways wanted to see some 64-bits benchmarks too, thanks for the review

[deanjo]

@Michael :

Now, as Ubuntu 8.10 and Fedora 10 are marked stable, this would be a good time to rerun the tests with Ubuntu and Fedora (7.x, 8.x, etc.) on the same hardware as the "Fedora 10 vs. Ubuntu 8.10"-test

If you would wait 20 days you can include openSUSE 11.1 in that too.

[glasen]

I think the more interesting comparison is with the series of tests which showed Ubuntu's performance decline very sharply after 7.04 and recover a little with 8.10. The fact that Fedora 10 and Ubuntu 8.10 are in effect identical performers leads me to wonder if all desktop distributions have suffered a big performance hit after kernel 2.6.15 (the Ubuntu 7.04 kernel).

Sorry, but you are only the 100000st person who believes that the test Phoronix had run with Ubuntu 7.04, 7.10 etc. were correct. There is enough prove that something went wrong during the tests (e.g. my P3-1000Mhz gets nearly the same numbers for Ubuntu 8.10 as the tested Core2Duo 1,87Ghz and my P-Mobile 1,7GHz gets nearly the same numbers for Ubuntu 8.10 as Ubuntu 7.04 in the Phoronix test, etc), so the numbers for the old tests can't be trusted.

@Michael :

Now, as Ubuntu 8.10 and Fedora 10 are marked stable, this would be a good time to rerun the tests with Ubuntu and Fedora (7.x, 8.x, etc.) on the same hardware as the "Fedora 10 vs. Ubuntu 8.10"-test

[deanjo]

That's really what these benchmarks should be about, is are there speed differences with different kernels, and how does different software compare. On the Mac tests, were they using completely different libraries or were they similar or the same. It's just really really hard to do any kind of fair comparisons, and harder to try to find out what is causing the performance differences. Those are the things that are fun to look at for me.

But, Phoronix still does good by pointing out the end results of everything put together in these software bundles aka distrobutions. That's "real world". But if there are different libraries or software users can switch to that are faster or whatnot, those things should be noted, since you should be able to install whatever the hell you want on your computer.

OS vs OS comparisons should really be done utilizing the distro's packages anyhows as that is how most people run those packages. In a hardware comparison test then the same version of OS should be used and the packages recompiled.

[Yfrwlf]

Maybe just newer version of some apps/libraries etc. etc. included in newer versions of Ubuntu and other distros "do things" other way then before? For example: lame compress files better then before even with the same options - it's clearly theoretically and stupid example. I have low end PC and new Ubuntu is faster then previous. And Mac OS was faster in Phoronix benchmark, so it's something wrong with those tests :>

That's really what these benchmarks should be about, is are there speed differences with different kernels, and how does different software compare. On the Mac tests, were they using completely different libraries or were they similar or the same. It's just really really hard to do any kind of fair comparisons, and harder to try to find out what is causing the performance differences. Those are the things that are fun to look at for me.

But, Phoronix still does good by pointing out the end results of everything put together in these software bundles aka distrobutions. That's "real world". But if there are different libraries or software users can switch to that are faster or whatnot, those things should be noted, since you should be able to install whatever the hell you want on your computer.

[kraftman]

I think the more interesting comparison is with the series of tests which showed Ubuntu's performance decline very sharply after 7.04 and recover a little with 8.10. The fact that Fedora 10 and Ubuntu 8.10 are in effect identical performers leads me to wonder if all desktop distributions have suffered a big performance hit after kernel 2.6.15 (the Ubuntu 7.04 kernel). Maybe Ubuntu 7.04 was by some chance simply vastly better than all other distros at the time but I doubt it. Another factor is the Desktop Environment. Maybe recent versions of Gnome are simply sucking the life out of systems...again unlikely and it doesn't reflect my experience with Gnome, KDE, Xfce, Fluxbox, Openbox etc. I'm inclined to think that Phoronix's tests are illustrating a general and severe decline in performance of the Linux kernel based desktop.

Maybe just newer version of some apps/libraries etc. etc. included in newer versions of Ubuntu and other distros "do things" other way then before? For example: lame compress files better then before even with the same options - it's clearly theoretically and stupid example. I have low end PC and new Ubuntu is faster then previous. And Mac OS was faster in Phoronix benchmark, so it's something wrong with those tests :>

[Takla]

I think the more interesting comparison is with the series of tests which showed Ubuntu's performance decline very sharply after 7.04 and recover a little with 8.10. The fact that Fedora 10 and Ubuntu 8.10 are in effect identical performers leads me to wonder if all desktop distributions have suffered a big performance hit after kernel 2.6.15 (the Ubuntu 7.04 kernel). Maybe Ubuntu 7.04 was by some chance simply vastly better than all other distros at the time but I doubt it. Another factor is the Desktop Environment. Maybe recent versions of Gnome are simply sucking the life out of systems...again unlikely and it doesn't reflect my experience with Gnome, KDE, Xfce, Fluxbox, Openbox etc. I'm inclined to think that Phoronix's tests are illustrating a general and severe decline in performance of the Linux kernel based desktop.

[Sacha]

I think the most interesting conclusion is 64-bit distros run the OpenSSL test more than twice as fast as 32-bit distros. I didn't know the gap was so big anyway. Not sure how it helps my every day activities though.