Again these issues are easily dealt with a build service where optimized packages can be built for pretty much any arch. openSUSE's public build service for example is pretty much next day releases when dealing with items such as X11, kernel, alsa, gcc and the various other quick changing projects.
That's the thing, I don't know where you see these huge differences. As far as the game tests in the above article goes again this can be done on any distro if you wish to compile from scratch or setup a build service to do so automatically. At work for example I run a build server there and separate packages are built for pentium 4, amd64, ia32e and ppc (distributed among various machines, some old workstations to newer machines that are just not used alot using icecream) . When a update comes out from the update service (or a upgrade version from watched projects) it goes out and retrieves the src rpm and builds updates for all of them utilizing the specific optimizations. Set and forget.
You said that "Bare minimum services running are just as easily done on other distro's". So if I install Ubuntu and disable the running services, I will see a speed equal to Gentoo minus the 3% gcc-flags gains?
I have done it, but it still was much more than 3% slower.