AMD Makes A Compelling Case For Budget-Friendly Ryzen Dedicated Servers

[Hefferbub]

I built a server like this about a year and a half ago. It has performed beautifully hosting server VMs through Proxmox and was an excellent value compared to traditional server offerings. It is cool, quiet and fast for my relatively modest workload, while being quite inexpensive.

What made this motherboard especially valuable was its support for PCIE bifurcation on it’s 16X slot, so for $80 I could add 4 PCIE GEN4 M2 SSD slots, each with a full 4 lanes. And since it has basic built in video support, I didn’t need that slot for a video card.

Case Antec P101 Silent $110
Motherboard ASRock Rack X570D4U-2L2T $536
CPU Ryzen 9 3650X 16/32 ZEN2 $767
Memory 4X Kingston 32GB DDR4 ECC UDIMM P/N: KSM32ED8/32ME $644
CPU & case fans Noctua NH-U12S SE-AM4 + 3 case fans $141.72
SSD Card NVME x16 to quad X4 Asus HYPER M.2 X16 GEN 4 CARD $80
OS SSD Sabrent 512GB Rocket GEN3 (SB-ROCKET-512) 800TBW $76
DATA SSDs 2X Addlink S90 2TB GEN4 NVME (3600TBW endurance) $745 (with room for 2 more)
IPMI Onboard ASpeed IPMI/BMI
Power Supply Seasonic (SSR-550PX) 80+ Platinum 550W S/N: R2007EA206400096 (10 yr) $120
Total cost ~$3500

[Michael]

Michael

>> and the WattsUp Pro external power meter no longer working with the latest Linux kernels due to some USB serial driver regression.

I don't understand why you didn't just use separate older/compat machine/box/tower to run the linux version that has a kernel that works with the power meter ?
then run a scp at the end of test to get the power log

Added latency with recording the data on a per-test basis, etc. Plus not worth the overall hassle.

[chithanh]

That's nice to know and confirms what I've read; about how it all comes down to whether or not the firmware will support those features.

To my knowledge, SME works an all Ryzen CPUs and on all (retail) AM4 mobos. The option to enable it is sometimes hidden in AMD CBS section of your UEFI setup. SEV is another question though, I understand that the CPU firmware that AMD releases on consumer platforms (including Threadripper) doesn't have SEV functions.
Also older Ryzen CPUs have hardware bugs that prevent SME from working as expected. There used to be conflict between SME and amdgpu (but not radeon or nouveau etc.) drivers, but I understand that this was worked around so it at least no longer fails non-gracefully. About APUs I don't know.
Perhaps agd5f can tell how it is exactly.

Theoretically one could use a DASH enabled motherboard + Ryzen PRO CPU to get the same oob-management "server" experience - and they come cheaper than the ASRockRacks with built-in ASpeed BMC.
Unfortunately the only motherboards with DASH that seem to be available are the ones with Realtek nics (eg. ASRock A520M-HDVP/DASH). It looks like DASH support under Linux seems to be an afterthought (or actively-hostile-thought) for both ASRock and Realtek. The latter don't publicly document their chips so its hard to know how to talk to the OOB engine there.

I understand that DASH support exists only for a variant of RTL8111 chipset so you don't get DASH mobos without Realtek LAN. But there are Linux tools.

https://developer.amd.com/tools-for-dmtf-dash/

[agd5f]

Also older Ryzen CPUs have hardware bugs that prevent SME from working as expected. There used to be conflict between SME and amdgpu (but not radeon or nouveau etc.) drivers, but I understand that this was worked around so it at least no longer fails non-gracefully. About APUs I don't know.
Perhaps agd5f can tell how it is exactly.

The display hardware on older APUs (pre-Renoir) requires a 1:1 mapping for latency reasons when accessing display buffers in system ram when using the IOMMU and the devices are limited to a 40 bit address space. Using SME requires a non-1:1 mapping since the encryption bit is above 40. The same thing applies to APUs supported with radeon or any other device or driver (i.e., driver doesn't use the DMA API properly) which has similar limitations. AMD dGPUs are not affected since they are stand alone PCI devices and the drivers use the DMA API correctly.

[coder]

At the moment the server motherboards seem to principally be ASRock Rack but I am told much more is on the way.

This is very encouraging to hear, because I've watched their X570 mini-ATX board, with 10 Gig Ethernet, go from "very pricey" to "OMG, that's a lot of money!", over the past year or so.

I would really like to upgrade my fileserver to a X570 server/workstation board + Ryzen 5700X. Admittedly, I could go a little lower on the CPU, but I like an option that gives me good versatility within a 65W power envelope.

One nice thing about a BMC chip is that it means I don't need to add a dGPU. And you can't just go with an APU if you want ECC memory.

I originally built this fileserver with a $165 ASUS AM2+ server board and a $120 Phenom II CPU. Not only did it support ECC memory, but the BIOS even supported patrol scrub, which slowly walks through memory and checks for parity errors. And, for the time, it was a little unusual to have 6 SATA 3 ports. It's been a great machine, these past dozen years or so, but as I upgraded the storage and NIC, I now feel like the CPU could be holding it back.

[trueblue]

any good source for Pro APUs? once got scamed by aliexpress "200GE Pro" ...

I bought a ryzen pro 5650g from this vendor (www.groves-inc.com) in January. It cost about $310. The only problem is that it took about 3-4 weeks to get from Germany to Tennessee. Not too bad for having to go through customs, and Covid was at its peak in the US. Here is the link for the 5750g.

https://www.grooves-inc.com/amd-ryzen-pro-5750g-ghz-kerne-threads-cache-speicher-socket-am4-oem-amd-hardware-electronic-pZZa1-2100415140.html?language=en&currency=USD&_z=us

Overall, I was happy with the experience, and the ordered price was equal to suggested retail + local taxes. They included all the shipping and custom fees in the price. I would have ordered from a US vendor but no one had it in stock to ship immediately. I went from a 2400g (non-pro) to the 5650g with ECC memory. The memory is detected as ecc by linux, but I think kernel 5.17 will be the best kernel to use with the pro 5000g series.

The graphics performance is quite adequate for my use case, as I am not a gamer. My only regret is that I did not go ahead and spring for the 5750g.

[coder]

I bought a ryzen pro 5650g from this vendor (www.groves-inc.com) in January. It cost about $310. The only problem is that it took about 3-4 weeks to get from Germany to Tennessee. Not too bad for having to go through customs, and Covid was at its peak in the US. Here is the link for the 5750g.

https://www.grooves-inc.com/amd-ryzen-pro-5750g-ghz-kerne-threads-cache-speicher-socket-am4-oem-amd-hardware-electronic-pZZa1-2100415140.html?language=en&currency=USD&_z=us

...

That's awesome! Thank you so much!

The only misgiving I have is when the 5600X was benchmarked against the 5600G, the X had a very consistent margin over the G. So, if I'm going to use a motherboard with a BMC, then I'll probably stick with an X-series.

[Yoshi]

I'd love to buy one of these boards, but in europe, only the non-10G version is available. And at exorbitant prices...

[deusexmachina]

Hopefully Project X succeeds!

[Turranx]

This talk about using Ryzen as a server platform both excites me and concerns me. My biggest concern is over the possible use of ECC RAM. Depending on where you look, you get different stories:

• https://www.cgdirector.com/do-you-need-ecc-memory/
• https://www.reddit.com/r/Amd/comment...n_ecc_support/
• ...and many more but I am too lazy to list here...

So when it came time to put together a solid workstation, I went with these parts:

• Asus Pro WS X570-ACE motherboard
• AMD Ryzen 7 PRO 3700 (100-000000073)
• v-Color 64GB (4 x 16GB) ECC DDR4 2666MHz (PC4-21300) VLP CL19 2Rx8 (TE416G26D819VK-VKC)

When I tried to overclock the RAM I began to see ECC error messages showing up in the Windows 10 System Event Log. Based on that I knew two things: 1) I had to drop the RAM speed back down, 2) ECC message reporting was working properly at all layers (RAM, mobo, CPU, OS).

From what I've read on the Internet, this RAM seems to be THE go-to sticks for a DIY Ryzen/Threadripper/EPYC ECC build: Kingston Server Premier 32GB 3200MHz DDR4 ECC CL22 DIMM 2Rx8