Benchmarking An ARM 96-Core Cavium ThunderX System

[coder]

I would assume a lack of memory bandwidth, or inefficient way for those 96 threads to access memory. I wonder how the CPU load was.

People don't think about cache coherency. ...with 96 cores, let me tell you: it matters.

I think that's a big reason why this thing manages to pull out pretty sizeable numbers in tasks with high data locality, and virtually falls on its face, elsewhere. Moar cores is not necessarily better. For memory-intensive workloads, you'd rather have more machines w/ fewer cores - even running at the same speed and with proportionally less memory bandwidth.

Speaking of which, what are the popular clustering solutions, these days? Whatever supplanted to the vaunted Beowulf clusters of yore? Like 15 years ago, you could hardly find a Linux-related thread on Slashdot where someone wouldn't mention them.

[GraysonPeddie]

Ah... I was just reading through the benchmarks without reading the entire article. Thanks.

[edwaleni]

Not even a Kill-A-Watt for the whole system? I mean, you can measure it at the wall.

I think Michael shared that he did not have physical access to the hardware, therefore any kind of power measurements weren't available. If the box he had access to was in a datacenter, it would be even less likely he could just plug in a kill-a-watt into the PDU.

Hopefully Cavium or Gigabyte will send him a ThunderX2 demo in the near future so he can test in more detail.

[GraysonPeddie]

Not even a Kill-A-Watt for the whole system? I mean, you can measure it at the wall.

[Michael]

What's the power consumption for 96-core ThunderX at idle and under load?

Unfortunately no idea since no physical access to be able to attach a power meter.

[GraysonPeddie]

What's the power consumption for 96-core ThunderX at idle and under load?

[Zola]

It's not disappointing because of its blatant lack of single-thread performance.

It's disappointing because Phoronix teed up benchmarks that should literally fall right into the lap of an OMG 96 COAR system with gobs of memory bandwidth and a pretty large power consumption envelope that's as least as high as a dual socket Xeon or Epyc system. And even there it's clearly inferior to a desktop system you could have bought last year, much less a real server.

It's very old ARM semi costume design based on the A57's which ware not efficient & counts as the worst ever ARM OoO design. Pump the things up to the A72 reference & you will get 20% more performance in 33% smaller DTP . Now that brings us down to 67W for a for a similar 48 core part. Then cut that again in half switching it to 16~12 nm FinFET & you get to the 33.5W per SoC or 67W for system containing two of them. Guess what now ARM server becomes 2x more efficient at least in worst case scenario while offering same to better performance in many task workloads as cloud storage and dynamic IO requests. 2x less energy = 2x cheaper ownership cost. Now I did only narrow it down to matching lithograph and wide spread design up to date. Arm has a unique current market advantage that it can push power advantage even future more by combination of OoO and in order core's which simply is not existent on X86 world. Now imagine a 100 core system (48 A72 based ones +2 A55 one's per socket) that in active idle state still full capable handling back & RT tasks uses 1W or less of power. That is an order of magnitude difference.

[eva2000]

Nice thanks for the benchmarks. I had started playing with 96 core Cavium ThunderX the other day too but ran into a road block on CentOS 7 my go to OS as there weren't any official MariaDB 10.x aarch64 YUM repo available yet. The benchmark numbers here leave a lot to be desired though. So while I am waiting on official MariaDB 10.x aarch64 YUM repo, been playing with AMD EPYC 7401P - very interesting beast for the price !

[directhex]

Has anyone seen benchmarks of this against A-series Opteron?

[stalkerg]

it's only 28nm....