Benchmarks Of The Gentoo-Based Calculate Linux

[pingufunkybeat]

You did notice that the article is not about Gentoo, or any other source distribution at all?

[pvtcupcakes]

There is an out of the box Gentoo, it's called a stage3 install.
However, you'd still need to configure the kernel. And no two kernels will be alike unless you use genkernel.

[allquixotic]

I kinda don't get why everyone seems to just agree that it doesn't make sense to let a stock Ubuntu run against a fully customized Gentoo. Does seem unfair of course, but could give us a good idea how big real life differences between those system really are.

How about benchmarking a stock Ubuntu (--> mainstream system), a fully optimized Gentoo (--> tweaker's system), a recent version of Fedora (--> bleeding edge binary system) and a Debian system configured for maximum stability? (Not just in terms of cpu performance, but also things like power consumption, RAM usage, disk usage and so on). I guess some of the results would't be as significant as some people like to believe...

Benchmarking an artificial CPU-bound program that gains enormous performance with SSE4, on a Core i7 system, exhibits a large performance delta. To add to the performance difference, you can turn on -flto and whole program optimization under GCC 4.5.x. If the benchmark program is unable to detect higher SSE extensions at runtime and enable inline assembly to use them, then it's up to GCC to codegen the SSE extensions at build-time. Unless you specify the right compile-time flags, then, it won't use any SSE4 extensions, because they might not run on some processors.

The performance difference between a CPU-bound program built against i686 (basically a Pentium 3) and a Core i7 (with SSE4), where the former lacks whole program optimization and the latter includes it, would be significant indeed. But your results only tell you about the performance delta for a synthetic benchmark. Unless you're into HPC, your own programs will be using a smattering of disk I/O, hammering virtual memory, interrupts to miscellaneous devices like graphics and keyboard input, network, and so on. A real program doesn't just sit there and run numbers cycle after cycle.

So, in fact, you would see very little performance delta for real programs that people actually find useful, except maybe in the case of math-intensive programs (big prime finders and such). First of all, math-intensive practical applications such as mplayer do detect SSE at runtime, and will enable the SSE code path (and MMX) even if the binary was compiled for i386. Second, most programs don't just calculate for very long periods of time without making some expensive system call: disk, network, graphics, or user input. The latency involved with any such request is so large that the time spent calculating is but a small fraction of the time spent waiting -- and because of hardware limitations, it is often impossible to reduce the latency of the system calls. These factors reduce the possible amount of performance optimization you can get from SSE, -flto, or generally anything that reduces the computational complexity of the program.

Seeing how the most common (and commonly-touted) speed-up applied in Gentoo is this exact sort of compile-time optimization, I would expect that such tests would reveal that artificial benchmarks run nicely on Gentoo, but real programs run about the same. If Gentoo were seriously focused on performance, it would investigate ways to shave off nanoseconds from the latencies incurred in device I/O, or implement optimizations in common programs that eliminate or coalesce device I/O operations as much as possible. It would be infinitely more valuable to Free Software to have folks who care about performance learn a bit about programming and then contribute smart optimizations to desktop software. These optimizations would benefit all distros, regardless of how the program is compiled, and the magnitude of these optimizations would be orders higher than anything Gentoo can come up with by tweaking CFLAGS.

Don't believe for a second that the free desktop is already optimized. It's far from it. Most graphical applications have been engineered for maintainability and correctness; once these two goals are achieved, the developers consider their work done. But all the I/O done by these programs can be re-arranged, re-scheduled, and minimized in ways that would make GNOME, KDE, or any other desktop environment run a lot faster (more responsive) on all hardware. Anyone who has tried to port existing Free Software from the desktop to an embedded Linux environment knows, that it's fairly easy to look at an existing piece of software and point out a few dozen places where it could be optimized, without delving into assembly or SSE or anything like that.

But these are the real optimizations, not just a casual CFLAG. Making this type of optimization requires real engineering knowledge, rather than applying a blanket solution to all your software. The only piece of software that even has a chance of automating this sort of optimization is the compiler, but unfortunately, many of these situations are difficult to make the compiler detect. Research done by big companies like Coverity is slowly improving the robustness of our tools that check for correctness and assist with optimization, but for now, what can be done with hand optimization is still decades ahead of automated tools.

However:

1. I understand and accept that at least some Gentoo users (especially ones posting here) don't claim that Gentoo's main advantage is performance. Some have even conceded that the performance advantage is non-existent. For those Gentoo users who still hold that Gentoo's primary platform feature is performance, I think the onus is now on them to show us quantitatively that this is the case with real programs.

2. Despite the fact that I have advanced an argument that I think Gentoo's performance delta is not significant for real apps, I would still welcome any testing done by Michael or anyone else in the community, whether that testing supports my conclusion or not. I'm fine with being proven wrong; what I am offering here are a few engineering reasons for believing that any such tests would lean a certain way. So by all means, optimize a Gentoo install as much as you can, and benchmark it. I'll add my voice to the "let's see the numbers please" crowd.

[brent]

[daveerickson]

What about setting up a list of benchmarks, then tweaking Gentoo to get the best possible results for that one task.

The system would literally be a bare install with only the one package to benchmark and the benchmarking software, presumably PTS. The kernel could be tweaked with each permutation being benchmarked. Same with USE flags, and CFLAGS. I know, there are TONS of permutations, but isn't that something that PTS can do with the kernel?

Then we could get actual "best performance possible" configurations for a given task with Gentoo. A person could also see how that performance stacks up to their configuration or a binary distro. Then a determination about whether it is worth the extra effort to use Gentoo for a given task or not. Seems like it could end up being very valuable info.

[RealNC]

What are people arguing about? It is a benchmark of Calc, not Gentoo.

Take it for what it is, seriously. Yes, Michael for some reason mentions "Gentoo" in the title while he never mentions "Debian" in titles that have Ubuntu in them, but that's how Michael is; inconsistent.

[Arch]

Then we could get actual "best performance possible" configurations for a given task with Gentoo. A person could also see how that performance stacks up to their configuration or a binary distro. Then a determination about whether it is worth the extra effort to use Gentoo for a given task or not. Seems like it could end up being very valuable info.

My point exactly. I think noone would question that a system can be optimized for certain tasks and that certain distributions lend themselves more to this kind of tweaking ... the question is whether it's worth the effort. (A question that tends to produce big discussions, but seldomly any hard numbers)

Tackling these kinds of issues should make for a few interesting articles.

[yotambien]

Which distro are you running, yotambien?

I would add the excellent documentation and community to Gentoo's strengths. And the compilation itself is automated -- that's the whole point.

I run Debian with testing, unstable and experimental repositories. Why?

I agree about the documentation. Whenever I have a problem I insert a generic search in Google such as "Linux + problem", and more often than not I end up finding the solution out of some Gentoo forum or wiki. But to be fair, a good many times I also find good information from the Ubuntu wiki. Only when I don't find anything I specifically include "Debian" in the search pattern. Usually everything is out there one way or another.

[clavko]

...I'm a bit underwhelmed by these benchmarks. So I'm wondering perhaps Michael didn't use the right CFLAGS. Also I'm wondering if the wise men from the Gentoo community (all 4 of them ) have reached their verdict and are declaring this benchmark flawed.

I don't get it. Calc linux is obviously a binary distro. The main reason why Gentoo might have some performance differences is related to compiling software directly on your machine. If Michael changed CFLAGS nothing would of happened, as far as I see there wasn't any compiling involved, hence no need.

I'm not sure why Michael felt the pressure to dig out some obscure, but "Gentoo based, oooh" binary distro, bench it out and plaster "Gentoo, Gentoo, Gentoo" all over the article. I wouldn't mind Phoronix never benchmark Gentoo per se at all, but this kind of behaviour is quite interesting, don't you think?

[Adarion]

Gentoo based? Hmm. But still they have to provide GENERIC x86 binaries and GENERIC USE flags suitable for a broad user base. So not a real Gentoo.
But benchmarking a real Gentoo would mean setting up a real Gentoo and that takes some time. Well, maybe for christmas.