AMD Ryzen Threadripper PRO 5965WX Memory Scaling Benchmarks On Linux

[intelfx]

I know for my 5900X it has a frequency and timing wall with 128GB of RAM. I can't go above 3600MT/s, below a CAS of 14 cycles, and some timings need to be relaxed. If I remove two sticks and run 64GB I can run 4000MT/s.

What custom timings do you use with 128 gigs at 3600MT/s on the 5900X? What RAM kit and how fast were you able to go in the end?
I'm very curious as I've got the exact same build and I'm hitting the exactly same problems.

[Anux]

At one DIMM per channel, the commands would all be on different channels.

Ah yes I see. But still, selecting between different channels must come with a penality, the more channels you have to select the higher the latency?

The whole point of registered memory is to add an electrical buffer, to relieve the load on the memory controller. The downside is this usually adds about a cycle worth of latency and adds a tiny bit of cost per DIMM - that's why it's not done for consumer platforms. Therefore, the memory controller it's doing much less work per DIMM than with unbuffered.

If you refer to RDIMMs than only adressing and commands are buffered and that should mainly mitigate the load from high capacity DIMMs. But even LRDIMMs will load the mem controller, surely not as much as UDIMMs but it needs to communicate with the DIMMs which cost energy and the more DIMMs the more energy.

Apart from that I just noticed that Michael was using stanndard UDIMMs.

[coder]

Ah yes I see. But still, selecting between different channels must come with a penality, the more channels you have to select the higher the latency?

There's a switch fabric in the I/O die. I imagine it's hard-wired to support all 8 memory channels. So, would you gain anything by reducing that channel-count? I wouldn't assume so.

Apart from that I just noticed that Michael was using stanndard UDIMMs.

Good catch.

When I was fiddling around with an old Intel server, at work, I noticed the docs said it could support fewer and smaller UDIMMs than RDIMMs (IIRC, the max memory was only 24 or 48 GB if UDIMMs, but up to 288 if RDIMMs). So, maybe TR can support only 1 DIMM per channel, if UDIMMs? I think a lot of servers support only RDIMMs, so I was surprised to find that machine supported UDIMMs. I think it's more common for workstations to support both.

[Anux]

When I was fiddling around with an old Intel server, at work, I noticed the docs said it could support fewer and smaller UDIMMs than RDIMMs (IIRC, the max memory was only 24 or 48 GB if UDIMMs, but up to 288 if RDIMMs).

Don't make the error of thinking that those docs are technical correct. Most of the time those max mem specs are just what's aviable (or has been tested) at the time of writing. If later there are bigger UDIMMs it's more than likely they would also work.

RDIMM and UDIMM have the same pin layout and share many similarities in function. Most server chips (atleast from AMD since opteron) share the same die with desktop chips and therefore also the same memory controller. So support for R/UDIMMs and ECC mostly depends on firmware, microcode and BIOS/Board.
Edit: old Intels had their memory controller on the chipset, I think everything pre Core i *

But you probably know, market segmentation and reduction of support costs steals us all those features.

[coder]

Don't make the error of thinking that those docs are technical correct. Most of the time those max mem specs are just what's aviable (or has been tested) at the time of writing. If later there are bigger UDIMMs it's more than likely they would also work.

It had to do with the number of DIMMs-per-channel (DPC) that could be supported with UDIMM vs. RDIMM, as well as I think single-rank vs. dual-rank. I think it might've supported something like 2 DPC of SR UDIMMs, but only one DPC of DR UDIMMs. I remember they definitely talked about ranks.

Anyway, that was my point. If higher-capacity DIMMs became available since the docs were written, then presumably both the UDIMM and RDIMM max memory specs would increase proportionally (up to the limit of how many physical address bits the CPU actually supports).

Edit: old Intels had their memory controller on the chipset, I think everything pre Core i *

Yeah, back then, you could use ECC memory with any CPU. You just needed to buy a motherboard that supported it. Intel sort of brought that back, in the Alder Lake generation. If you buy a motherboard with a W680 chipset, you can use ECC memory with any Alder Lake CPU! Of course, it's completely artificial that they tied it to the motherboard chipset, since the chipset no longer mediates memory transactions (as you mentioned).