AMD Threadripper 2990WX Linux Benchmarks: The 32-Core / 64-Thread Beast

[haasn]

Re: THP and general performance on a NUMA system, I'd like to point out that openSuSE Tumbleweed defaults transparent hugepages to "always" in their kernel. Given the benchmarks we've seen from various linux distros (summary: tumbleweed is nearly always ahead of the rest) in the other article, it certainly can't be a crippling factor when it comes to performance. (Although it would be interesting to see dedicated benchmarks for THB on vs off on the threadrippers, as well as SLAB vs SLUB)

I'm also no longer sure if disabling NUMA node migration really helps. I think it helped initially, but after testing the system for a while longer the results degraded again. It's possible that I only initially got good results because the system was already balanced at that point in time, but this property doesn't get maintained with NUMA balancing disabled? I need to repeat this test properly by enabling or disabling NUMA balancing from the boot process onwards and testing under similar circumstances, but I don't have the ability to easily do that right now.

[chithanh]

Transparent hugepages on non-NUMA systems are almost always neutral or beneficial to performance. This is why most distros have them enabled.
On NUMA systems, the benefits can be reduced, or even reversed as the Usenix paper has shown.

[bridgman]

haasn thanks for doing some testing - mostly was interested in knowing if NUMA page migration was one of the things we should be looking at. Seems like the answer is "yes" even if it is not the full solution.

I don't know if "Game mode" just interleaves the physical address space across DRAM channels and presents as a non-NUMA system or whether it disables cores as well to deal with Windows-isms... will try to find out.

If it only interleaves and disables NUMA but does not disable cores then it might be another interesting option, assuming it is available on Linux (it was supposed to be).

Maybe we should take this to the kernel.org bug tracker, now that we know a linux setting can influence the behaviour (and therefore linux might be able to do something about it)?

Agreed. Our kernel devs on the CPU side hang out there so seems like the right place.

I had previously played around with switching this setting between madvise and always (taking a page from openSuSe Tumbleweed's book here), but didn't notice an improvement (or degradation) in the observed behavior. I tried disabling it completely but also didn't notice any difference; do I need to reboot my system to properly test the effects of that change, or is it only relevant for new processes?

My understanding was "took effect immediately" but I don't think I have ever confirmed that. If I can find a bit of free time will try to pick through the kernel code and find out for sure.

[madscientist159]

Looks like a nice powerful rental. Not sure that's interesting. AMD retains full control of these boxes via the PSP, but wants you (the consumer) to pay when anything breaks. How is that ethical on any level?

If it's a rental, they should be paying for electricity and maintenance. If it's not a rental, they shouldn't be retaining superuser access to "your" machine.

[bridgman]

Hi Tim

Just curious, how does AMD retain control without a network interface ? Would it help if the motherboard was wrapped in tinfoil ?

Seriously, I really like what you guys are doing with the TALOS products, but it's probably time to ask Michael to put some affiliation info in your user profile.

[haasn]

Agreed. Our kernel devs on the CPU side hang out there so seems like the right place..

I decided to move this discussion there: https://bugzilla.kernel.org/show_bug.cgi?id=200811

Feel free to respond to that ticket if you have any further input or ideas.

[madscientist159]

Hi Tim

Just curious, how does AMD retain control without a network interface ? Would it help if the motherboard was wrapped in tinfoil ?

Seriously, I really like what you guys are doing with the TALOS products, but it's probably time to ask Michael to put some affiliation info in your user profile.

Heh, sure, no problem there. I normally mention affiliation, but since this is a personal issue I have with the PSP, didn't do so here.

As far as network goes, the PSP is a rather large black box, so you tell me what access channels might be in there. I would have no way of knowing, which is one of the fundamental problems here. AMD compatible mainboards usually ship with network ports, one could reasonably conjecture hooks provided by the mainboard vendor if needed. Since that's another large black box (UEFI, PI, etc.) again no way to know.

What I do know, however, is that there is a large binary-only (opaque) chunk of code that always runs, and runs at a very high privilege level. I also know that in addition to not knowing what's in it, that I cannot replace it because AMD keeps control of it via hardware-enforced signatures. I don't know if it will stop working years in the future, or what happens if a security flaw is found in it after AMD has stopped support. This is all highly concerning and my main point of contention with AMD's design choices. I understand why they were made, but do not agree with the monolithic approach where this is required for every single AMD product.

I like what you guys did with Vega and the open stack. Would love to have an owner controllable Vega type compute card with no DRM blocks and open firmware, as you know -- I'd very much like to discuss some possible product opportunities there, but I also note that the PSP is even in Vega Pro now, so not sure what can be done.

Please feel free to drop me a line over Email if you want to talk further a bit more in private.😁

[illwieckz]

If this is the still the case on the newer threadrippers I can't seriously recommend them to anybody with a straight face, the CPUs have been riddled and riddled with bugs. (First…, then…, the…, the…, etc.)

It's fun to see how you assume there is a viable alternative out there that does not suffer from a very long list of horrible fails¹. ¯\_(ツ)_/¯

Telling people to chose anything but something based on a mental bias is just FUD.

_________ 
¹ or please name it, people need to know which product benefits from other's list of fail because just telling people things are wrong and not giving solution is unable to help anybody.

[Tomin]

Chromium compile test is really a mess and not sure why they use it besides compiling something that at least Windows users know what it is.

Yes. It probably isn't a good choice then. Latest that I've compiled is coreboot but that's hardly suitable. Something I usually compile myself on any computer is Tensorflow. That's because the packaging in PyPI doesn't use AVX instructions but warns loudly that they could be used and it helps quite a bit (unless you are using GPU). I'm not sure if it's still too small but it's not very hard to compile. Last time I compiled it was on some older laptop i7 (Ivy Bridge I think) and it did take some time but these machines are much more powerful.

[zoomblab]

TR 2990WX destroyed the competition in all benchmarks here. Meanwhile, over at Anandtech they show a completely different picture.