AMD Navi "Blockchain" Card Support Being Added To Linux 5.10

[Jumbotron]

I'm not sure how much sense it makes to include the previous Fusion/HSA work. That was pretty Windows-centric and designed around completely different hardware - APUs with 48-bit addressing (same as the CPU) and using IOMMUv2 for transparent GPU access to unpinned CPU memory via recoverable page faults.

The dGPUs of the time were much more limited - 40 bit addressing, GPUVM and only able to access pinned memory that had been explicitly mapped to GPU by the driver. Discrete GPUs (ours and our competitors) are only now catching up with the APU capabilities we had in 2014.

We were able to re-use some of the earlier code - compiler front end, runtime and kernel driver skeleton mostly - but most of what we consider the ROCm stack today was written from scratch including compiler back end, memory management and of course all the libraries.

" Discrete GPUs (ours and our competitors) are only now catching up with the APU capabilities we had in 2014. "

Whoah...I just now caught that !! Really ??

Ok....so. A: When do you anticipate that AMD dGPUs will ACTUALLY catch up to your APU's of 2014 and in what way?

And.........B: What does this entail for AMD's APU's going forward particularly the upcoming Zen3 / RDNA 2 based "Cezanne" APU's ?

Oh...let me add C: Does Zen 4 / Genoa / 3rd Gen Infinity Architecture herald this "catching up with the APU capabilities we had in 2014" ?

[bridgman]

Ok....so. A: When do you anticipate that AMD dGPUs will ACTUALLY catch up to your APU's of 2014 and in what way?

And.........B: What does this entail for AMD's APU's going forward particularly the upcoming Zen3 / RDNA 2 based "Cezanne" APU's ?

Oh...let me add C: Does Zen 4 / Genoa / 3rd Gen Infinity Architecture herald this "catching up with the APU capabilities we had in 2014" ?

A: probably mid-2021 as we finish integrating HMM into our compute products and get the same kind of GPU access to unpinned OS-allocated memory

B: we actually took iGPUs a step back in the short term, using GPUVM code paths rather than ATC/IOMMUv2 to let them coexist with and be compatible with our dGPUs. I think we started this with Picasso and may do the same with Raven Ridge. Should make it easier to maintain consistent support across the entire stack and allow a developer working on a laptop to run the same code on a server without issues

C: yes, that should be the final missing piece of the solution

[Jumbotron]

Well....I'll be !! That's incredible info !!

I had forgotten to mention the HMM work being done in kernel and how that would play with you guys. Interesting to see about the "step back" in current APU iGPUs but quite understandable.

And on C: LOL....I get it. That's future plans and that's the domain of the suits...I get it. However, this little write up on Anandtech.com earlier this year after AMD's Financial Analyst Day had me thinking that by 2022 with Zen4 / Genoa/ 3rd Gen IA that finally all the hardware, memory, interconnects, etc. would be in place to finally fulfill the "Fusion" vision AMD set forth with APU's back in 2011 on a platform that did not have APU's.


" Another element to AMD’s Financial Analyst Day 2020 was the disclosure of how the company intends to evolve its interconnect strategy with its Infinity Fabric (IF). The plan over the next two generations of products is for the IF to turn into its own architectural design, no longer just between CPU-to-CPU or GPU-to-GPU, and future products will see a near all-to-all integration.

AMD introduced its Infinity Fabric with the first generation of Zen products, which was loosely described as a superset of Hypertransport allowing for fast connectivity between different chiplets within AMD’s enterprise processors, as well as between sockets in a multi-socket server. With Rome and Zen 2, the company unveiled its second generation IF, providing some more speed but also GPU-to-GPU connectivity.



This second generation design allowed two CPUs to be connected, as well as four GPUs to be connected in a ring, however the CPU-to-GPU connection was still based in PCIe. With the next generation, now dubbed Infinity Architecture, the company is scaling it not only to allow for an almost all-to-all connection (6 links per GPU) for up to eight GPUs, but also for CPU-to-GPU connectivity. This should allow for a magnitude of improved operation between the two, such as unified memory spaces and the benefits to come with that. AMD is citing a considerable performance uplift with this paradigm. "

Of course...IA becomes even more interesting with AMD's acquisition of Xilinx. And also that more exciting in my humble opionion. Cheers !

[andrei_me]

I just want to thanks Michael for Phoronix exists and we have access to developers ( in this case, bridgman ) so we can ask questions direct to them and have awesome answers

[Jumbotron]

You know...HOLY CRAP....it just hit me !!

Looking at that slide above again and ruminating on bridgman's wonderful insights and on AMD's recent acquisition of Xilinx.....it just hit me.

AMD by 2022 is going to be able to once again sell their own branded datacenter in a box as they did briefly after their acquisition of SeaMicro in 2012 and then dumped in 2015 after Lisa Su came on board. But...interestingly....AMD said they were "keeping the Freedom Fabric" that SeaMicro developed for their 10U datacenter in a box.

The whole reason AMD bought SeaMicro was to get a foothold at the time in the "ultra-dense" server market. This is the market that needed 1000+ VM capability in a
small-ish footprint.

And now...with Zen 4/ Genoa / Infinity Architecture ( how much of SeaMicro's "Freedom Fabric is engineered into Infinity Architecture ?? ) AMD will have the capability of selling a 4U box capable of 10,000 VMs. A REALLY big datacenter in a box...and a much smaller box than SeaMicro's 10U sized box.

And what makes that "datacenter in a box" a reality? Xilinx FPGA SmartNICs. Xilinx FPGA SmartNICs tied into AMD's Infinity Architecture with all to all cache coherent connectivity to CPU's, GPU's and FPGA's.

Whooooohhh BOY.....2022. CAN'T WAIT !!

[Jumbotron]

I just want to thanks Michael for Phoronix exists and we have access to developers ( in this case, bridgman ) so we can ask questions direct to them and have awesome answers

Truer words have rarely been written. It is a blessing. All the more reason to support Michael and his endeavors. With a tip, or better yet a subscription. I need to renew mine...but right now...Covid and a months long lay off have squeezed me dry. But I am trying to scrape up some dough for a sizable tip in the mean time when I can go ahead and make a lifetime sub payment. Hopefully after next years tax return + additional Stimulus + a new job.

[bridgman]

how much of SeaMicro's "Freedom Fabric is engineered into Infinity Architecture ??!

Not much at this point, unfortunately. Freedom Fabric was a large scale interconnect based on 1 and 10 gigabit Ethernet, while Infinity Fabric is more like an advanced version of Hyper Transport, connecting a smaller number of nodes but with much higher data rates and cache coherency.

[Jumbotron]

Not much at this point, unfortunately. Freedom Fabric was a large scale interconnect based on 1 and 10 gigabit Ethernet, while Infinity Fabric is more like an advanced version of Hyper Transport, connecting a smaller number of nodes but with much higher data rates and cache coherency.

So am I reading you right in mentioning gigabit ethernet and Freedom Fabric that ...

A: Freedom Fabric was an ethernet based interconnect to connect the various blades to one another inside the box
Or B: It was the Interconnect to multiple SeaMicro boxes....roughly analogous to Gen-Z ?
Or C: Both

And while I'm at it talking about interconnects. Things were getting very crowded after it took so long to get PCIe4 off the ground. We had CXL from Intel, OpenCAPI from IBM, CCIX from AMD, ARM and Xilinx and Gen-Z. Not to mention Infinity Architecture from you guys.

Now it looks like it's going to be CXL as a superset of PCIe 5 for internals and Gen-Z to tie together racks. That leaves CCIX and Infinity Fabric. What happens to CCIX now that Xilinx is in the house of AMD and Infinity Fabric? And what about CXL and Infinity Fabric ?

[bridgman]

So am I reading you right in mentioning gigabit ethernet and Freedom Fabric that ...

A: Freedom Fabric was an ethernet based interconnect to connect the various blades to one another inside the box
Or B: It was the Interconnect to multiple SeaMicro boxes....roughly analogous to Gen-Z ?
Or C: Both

I was never sure exactly where the lines were drawn re: the definition of Freedom Fabric, but IIRC it included:

- a 2D torus interconnect at 1 Gb/s between blades in a box (some reports say this was 2-lane PCIE rather than GbE)
- conventional 10 Gb/s interconnect outside the box (10 GbE)
- management SW that bridged across multiple boxes and did things like bandwidth allocation & QOS using FPGAs on both 1 Gb and 10 Gb NICs

Treat the above as low quality information but it's probably pretty close.

And while I'm at it talking about interconnects. Things were getting very crowded after it took so long to get PCIe4 off the ground. We had CXL from Intel, OpenCAPI from IBM, CCIX from AMD, ARM and Xilinx and Gen-Z. Not to mention Infinity Architecture from you guys.

Now it looks like it's going to be CXL as a superset of PCIe 5 for internals and Gen-Z to tie together racks. That leaves CCIX and Infinity Fabric. What happens to CCIX now that Xilinx is in the house of AMD and Infinity Fabric? And what about CXL and Infinity Fabric ?

I'm going to have to "no comment" that one.

[MadeUpName]

A big thanks to Jumbotron and Bridgman for really bringing it today. I am blown away by how many dots got connected for me in this thread.

Going back to some of the early comments about AMD not investing in software. AMD is crushing it on the CPU side right now and I can't imagine them no continuing to take market share from Intel. At the same time NVidia brough great graphics cards but can't deliver them. If AMDs new graphics cards turn out to be as good as claimed and they can actually supply them AMD is going to have to hire people just the shovel the money. That solves part of the problem, the other side is graphics engineers don't exactly grow on trees.