A Look At The Open-Source Talos II POWER9 Performance Against x86_64 Server CPUs

[zanny]

Pretty odd mix of very good and very bad results.

It looks like a lot of the really offensive losses for POWER are in encoders that often use hand optimized x86 ASM for vectorization. If there isn't a similar POWER ASM module available it would then probably fall back on a generic "safe" C version that doesn't vectorize as well if at all.

[Guest]

Looks like the POWER9 results aren't as good as they should be, given the hardware and price. Hopefully it's just a matter of tuning the performance in compilers, runtime, C library, kernel etc.

AMD's server CPU results are very impressive! They beat out the Intel CPUs most of the time, and seem to be the best option overall. (not sure about the cost though)

[tildearrow]

I have a question. Why are the encoding benchmark in seconds rather than speed (e.g. frames per second or speed compared to real-time)?

[Vistaus]

What am I supposed to do to get an invoice to my company when buying from the Talos website? There is not way to insert a VAT ID and they just don't answer emails...

... give them a call?

[bpetty]

Being a fully open sourced system, I think it did pretty well. Many other things resemble this, like AMD graphics drivers. The only difference is that for software its all free, but hardware has a real cost associated with it (especially if it is basically all custom, made in america). Even the cost isn't that bad depending on your perspective. Mass produced hardware built in low wage countries that have a ton of competition and years of development revisions will always have an edge. I am honestly surprised it performed this well, especially since many of those tests were probably not optimized for POWER as stated above. Improve the software by optimizing for this architecture and I bet things would look better... but you have to have the hardware before you can really start optimizing the software. I bet in 7 to 10 years it could be great in terms of price and performance. That may seem like a long time... but just look at Wayland (jk, err, not really).

[kgardas]

Michael, thanks for testing. Bellow I'd like to throw my own findings about the results I don't think are right or there is a need to remark something:

1) GNU MPC -- not only POWER9 is the slowest, but also Xeon Silver 4108 is faster than Dual AMD 7601? This does not look good. Either benchmark is AVX512 or whatever Intel prefers for vector computation or something is really strange with the benchmark.

2) Numpy -- isn't this python heavily hurt by POWER Spectre workarounds while those workarounds are not switched on on either of your EPYC/Xeon servers?

3) Timed Apache Compilation -- benchmark clearly does not scale to this number of cores/threads.

4) Timed Image Magic compilation -- scales much better than Apache, but still not so well. Look 28s for single 7601 and 23 for dual? So second CPU speeds just 5s instead of to half which would be done in well scalable bench.

5) Timed LLVM Compilation -- this scales even better than ImageMagic compilation

6) NodeJS/AOBench -- are those purely single-threaded benchmarks? Looks so to me and looks like POWER9 still needs some optimization since 3GHz EPYC/Xeon is faster than 3.8GHz POWER9 although it should be reversed.

7) PyBench/PHPBench -- both probably hurt by Spectre workaround on by default on POWER yet probably switched off on your EPYC/XEON systems?

[milkylainen]

I didn't include any due to difficult to determine reasonable values.... I could do them simply based upon retail CPU pricing, but then that would be skewed due to Talos II motherboard being more costly than x86_64 motherboards. Or I could include the particular motherboards + CPU pricing, but the motherboards used is somewhat arbitrary based upon what I have / was sent out. So particularly in the x86_64 space the TCO can vary quite a bit based upon motherboard and other components.

I think CPU pricing only is a fair metric for early systems like this. Motherboard manufacturing costs will drop fast if volume picks up.

[edwaleni]

Based on some of the early results Michael had done remotely, we knew the POWER9 was not going to perform well on codecs and media due to lack of VSX (former Altivec) support.

This was being addressed to some degree in the Chromium on POWER effort and IBM is well aware of it.

On the lzbench using libdeflate, I went back and looked up the source and it was heavily optimized for x86. So no surprise that it performed poorly.

At least now we have a baseline by which to measure the expected enhancements yet to come on Linux/POWER9.

With Talos leading the way on the hardware side, a new POWER ecosystem can begin to evolve as libraries and compilers take advantage of the capabilities.

I would really like to see an OpenCL test with POWER9 and the NVidia Volta, but in the case with Talos, a recent GeForce will do.

[V1tol]

6) NodeJS/AOBench -- are those purely single-threaded benchmarks?

Yes they are. First one is single-threaded server side Javascript. Second one is kind of raytracing bench that has single-threaded C and WebGL versions.

[darkbasic]

... give them a call?

They don't answer to emails, that's the problem. Do you have a phone number or any other kind of contact?
I've already placed my order but I just can't pay until they fix the billing.