However, It doesn't apply to design phase and compile time optimisations. For eg, ubuntu is compiled to support many old archetectures back to (i think) the original pentium and therefor does not allow a lot of optimisations that could be applied to a single archetecture system. Even if (like SSE, SSE2, etc optimisations) they can be detected and used at runtime, there is still the dead weight of the legacy support code in there takeing up memory space. Some distros build for i386, i think ubuntu doesn't go that far back, only to the i586.
This is why an well tuned gentoo system SHOULD run faster, load quicker and be lighter on memory usage than a system that is compiled to make sure it will work on every 32bit x86 archetecture you throw at it.
I guess the X86-64 build SHOULD dodge most of this legacy junk, if done properly?