8 vs. 12 Channel DDR5-6000 Memory Performance With AMD 5th Gen EPYC

[phoronix]

Phoronix: 8 vs. 12 Channel DDR5-6000 Memory Performance With AMD 5th Gen EPYC

As I wrote about last week within the Supermicro H13SSL-N EPYC Turin motherboard review, one of the factors leading me to purchasing that EPYC 9005 series motherboard was that this board offered support for full 12 channel DDR5-6000 memory performance compared to some of the other lower-cost Socket SP5 motherboards offering just 8 memory channels. For those wanting to quantify the performance difference between eight and twelve memory channels with AMD EPYC 9005 "Turin" processors, here are some benchmarks for showing the workloads that can really benefit from all 12 memory channels and other workloads where eight memory channels can be largely sufficient if looking to minimize costs.

https://www.phoronix.com/review/8-12-channel-epyc-9005

[fairydreaming]

What BIOS NUMA settings did you use? The STREAM benchmark results (355858.4 MB/s for ADD, 348347.3 MB/s for TRIAD) don't look very good considering that "per socket" Epyc Turin bandwidth advertised by AMD is 576 GB/s. Are you sure that your memory actually runs on 6000 MT/s speed?

But I'm glad that you included STREAM benchmark results at all, I miss them a lot in Phoronix tests.

[Michael]

What BIOS NUMA settings did you use? The STREAM benchmark results (355858.4 MB/s for ADD, 348347.3 MB/s for TRIAD) don't look very good considering that "per socket" Epyc Turin bandwidth advertised by AMD is 576 GB/s. Are you sure that your memory actually runs on 6000 MT/s speed?

But I'm glad that you included STREAM benchmark results at all, I miss them a lot in Phoronix tests.

All options were at their defaults. Keep in mind for STREAM, AMD does maintain their own downstream of STREAM that is optimized for Zen compared to the old upstream unmaintained STREAM. Yes, it was running at DDR5-6000.

[Eugeniu]

Very revealing. Thanks for sharing.

[JEBjames]

Michael

Interesting review.

Minor typo on page 1 "CPU core/thrad count" should be "thread"

[tchiwam]

Becomes interesting to compare dual socket @ 8 channels vs 1 socket @ 12 channels too for the same total amount of cores.

Thanks !

[coder]

What BIOS NUMA settings did you use? The STREAM benchmark results (355858.4 MB/s for ADD, 348347.3 MB/s for TRIAD) don't look very good considering that "per socket" Epyc Turin bandwidth advertised by AMD is 576 GB/s. Are you sure that your memory actually runs on 6000 MT/s speed?

Stream is one of those tricky benchmarks you'd want to understand, before interpreting the results. On x86 CPUs, the write-miss penalty means there's a phantom read that you should account for. So, the benchmark is only measuring 75% of the memory transactions that are actually happening. If we account for that, then ADD achieved 474.5 GB/s and TRIAD achieved 464.5 GB/s. I think that's reasonably close to what's practically achievable.

[S.Pam]

All options were at their defaults. Keep in mind for STREAM, AMD does maintain their own downstream of STREAM that is optimized for Zen compared to the old upstream unmaintained STREAM. Yes, it was running at DDR5-6000.

The numa settings can make a massive difference depending on workload. It would be nice to include some the variations in the benchmarks too.

[GPTshop.ai]

I decided to buy ASRock Rack BERGAMOD8-2L2T. It has only 8x DIMM slots but PCIe 5.0 M.2. I expect faster disk speeds to have a greater imact on performance than faster RAM speeds. Woud be great to have some benchmarks for this too...

[fairydreaming]

Stream is one of those tricky benchmarks you'd want to understand, before interpreting the results. On x86 CPUs, the write-miss penalty means there's a phantom read that you should account for. So, the benchmark is only measuring 75% of the memory transactions that are actually happening. If we account for that, then ADD achieved 474.5 GB/s and TRIAD achieved 464.5 GB/s. I think that's reasonably close to what's practically achievable.

Thanks for this information, very helpful. Based on what you said the highest possible STREAM TRIAD benchmark result:

• for Epyc Turin would be 75% * 576 GB/s = 432 GB/s
• for Epyc Genoa would be 75% * 460.8 GB/s = 345.6 GB/s.

But for example in Fujitsu Server PRIMERGY Performance Report for RX1440 M2 servers (Epyc Genoa) we can observe values close to 400 GB/s. Link to report here: https://sp.ts.fujitsu.com/dmsp/Publi...0-m2-ww-en.pdf

Any idea how is that possible?