Linux 5.14 To Allow Hot Unplug Of AMD Radeon GPUs

[phoronix]

Phoronix: Linux 5.14 To Allow Hot Unplug Of AMD Radeon GPUs

Linux 5.14 to debut later in the summer will allow for hot unplugging of AMD Radeon graphics cards such as when using an external GPU enclosure or passing back a GPU from a virtual machine to the host. Up until now the AMDGPU kernel driver hasn't cooperated nicely with the Radeon GPU for hot unplug events...

https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.14-AMDGPU-Hot-Unplug

[theriddick]

So this is FireWire stuff? there isn't a external PCIe solution is there? I've always wondered about the performance hit you'd take for having a external GPU. The bandwidth limitations use to be a big problem.

[Zan Lynx]

So this is FireWire stuff? there isn't a external PCIe solution is there? I've always wondered about the performance hit you'd take for having a external GPU. The bandwidth limitations use to be a big problem.

Thunderbolt and now USB-4 include external PCIe. That is what Thunderbolt is.

And Radeon GPUs are very popular in Thunderbolt eGPU cases because they are fully supported by Apple drivers.

[lyamc]

The biggest problem that I have with Thunderbolt enclosures is that you only have 4 lanes (40Gb/s) max, but generally I see closer to 2 lanes.

Add that with the additional latency caused by the run of the thunderbolt cable and you get some significant performance performance penalties.

Granted, even a 25% performance hit is better than having no GPU.

[mppix]

Add that with the additional latency caused by the run of the thunderbolt cable and you get some significant performance performance penalties.

??
If anything, added latency is due to redrivers but even that is marginal.
Sure, you are limited to 40Gb/s for now and we'll have to see how the usb-c clusterf_ evolves.

[oiaohm]

So this is FireWire stuff? there isn't a external PCIe solution is there? I've always wondered about the performance hit you'd take for having a external GPU. The bandwidth limitations use to be a big problem.

There is external PCIe solutions.

Most common is the thunderbolt 3 Apple/intel has. 4 lanes of PCI Express 3.0 max and lyamc sorry you are wrong that is only 32.4 Gbit/s not 40Gbit/s the total of the thunderbolt transfer system is 40Gb/s but you external thunderbolt devices don't get that ever. Also something to be horrible aware of some device with thunderbolt 3 have the lower power form that is only connect by 2 lanes of PCI Express to the computer on those yes seeing what Lyamc writes about only see like 2 lanes worth of data transfer is in fact tapping the amount of transfer out because you are in fact bottle-necked from computer pcie bus to the thunderbolt chip not thunderbolt 3 itself. This is not the only thing if you have connected a display and a external gpu on to the same thunderbolt port.

I know it with Nvidia but in 2021 CES ASUS demoed a external gpu "ASUS ROG XG Mobile, An External Graphics Dock For ROG Flow X13" This in fact has 8 lanes of PCI Express 3.0 done over a custom socket. There are other custom solutions out there that get 8 lanes of PCI Express to a external item.


This depends on what you are doing serous-ally. Those running gpus for particular compute like mining crypto currencies 1 lane PCI Expresss 3.0 is good enough in most cases.

Do note that lyamc says 25% hit. Horrible reality is some of the modern/current AMD GPUs have a 16x connector but only 8x lanes is really connected to anything. With GPU there seams to be a upper limit of what is usable bandwidth. Yes there are some desktop motherboards as well where you pci 16x slots are only 8x lane populated. So depending on what system you put this head to head with.

16x lanes pcie 3.0 or 8x lanes pci 4.0 amount of bandwidth seams important when you are wanting to render on one GPU and display on another or be streaming the output. So laptop with external GPU you should be considering external monitor as well.
8x lanes pcie 3.0 or 4x lanes pci 4.0 seams to be the point that 99% of your desktop applications/games with a monitor directly connected to the GPU will not be bottle-necked at all. About 70% of all applications will still not bottle-necked with the display output being rendered on a different GPU.
4x lanes pcie 3.0 what is full thunderbolt 3 this is good for about 70% of all applications without any bottleneck as long as the external GPU has a directly connected monitor. This drops to less than 10% if you attempt to render back on the laptop screen. Yes this is dependant that the only thing connected by the thunderbolt port controller is the GPU with thunderbolt 3. But even when the bottleneck of the display being back on the laptop screen you still get like 25%+ of the GPU performance. Thunderbolt 4 mandates that you get the 4x lanes of pcie 3.0 but application performance is the same.

2x lanes pcie Low Power thunderbolt 3 this one you are in trouble at under 30% of desktop applications will not manage to bottleneck it even with GPU connected direct to monitor and forget rendering back on laptop screen. There is just not enough bandwidth here to have display redirected back to screen and drive the GPU effectively.

The basics here even with compression of the output you are need between 2 to 3 lanes of pcie 3.0 to transfer the GPU generated display output without compression at all you will be spiking up to 4 lanes of pcie 3.0. There is a lot bandwidth need to transfer the output of a GPU over PCie.

There is not just one pcie external solution theriddick. I sorry there is no really clean answer.
External GPU some between
1) as good as internal(what is currently the custom vendor particular external GPU connections housing/cards...),
2) Performing better than laptop own gpu but under performing if you attempt todo particular things but otherwise generally good.
3) almost total crap this would be your thunderbolt 3.0 lower power and if you have put a thunderbolt 3.0 monitors with your external GPU on the same thunderbolt 3 port so in fact being bandwidth short.

Another horrible reality is I am not sure if we should care about apple users that much. Recent statement from apple developers says they are going down route that one day you will not be able to run Linux on their hardware anyway with the plans to remove external booting and writing to internal storage. Still supporting AMD GPU hot unplug will be a useful feature. This work with hotplug should also help in the case of fully needing to reset a GPU as well down the track so some of this work will be useful to us who don't use external GPUs in time.

[ssokolow]

Yes there are some desktop motherboards as well where you pci 16x slots are only 8x lane populated. So depending on what system you put this head to head with.

Wasn't that a mixture of these two factors I've seen in motherboards?

1. If you install an APU rather than a CPU, half the PCI-E lanes in x16 slot #1 will be reassigned.
2. PCI-E x16 slot #3 gets 8 lanes rather than 16, so populate slots 1 and 2 first.

[oiaohm]

Wasn't that a mixture of these two factors I've seen in motherboards?

1. If you install an APU rather than a CPU, half the PCI-E lanes in x16 slot #1 will be reassigned.
2. PCI-E x16 slot #3 gets 8 lanes rather than 16, so populate slots 1 and 2 first.

ssokolow I have not seen a APU as in being AMD CPU with integrated graphics do that one where the first slot has only 8x instead of 16x because you have a APU but I have seen a iGPU being a intel integrated graphics do that one. All the AMD APU I have seen your number of PCIe lanes from the CPU chip has not changed I could be wrong.

It does pay to be aware that AMD motherboards do support PCIE bifurcation. So the x16 slot closest to the CPU could be split with the next on the motherboard to provide to two x8 lanes PCI 4.0/3.0 with a B550 motherboard but this is not APU vs non APU cpu with amd. A 3 slot pci-e 16 B550 motherboard the first 2 are going to x8 PCI lanes each from the CPU and the last slot is going to be off the Chipset so only have x4 lanes at PCI 3.0 worth of performance max.

Slots 1 and 2 of PCI-E x16 on particular motherboards will only have x8 PCIe. Some of your older chipsets from Intel PCI-E slot one has 16x while only it is populated but when you populate the second PCI-E slot it auto performs bifurcation reducing both to x8 PCIe. Yes some of the intel iGPU you half you first 16x slot and it results in the second pci-e x16 slot on some motherboards now not having a CPU connection at all and being a useless slot.

There are quite a few cases where what looks to be a PCIe x16 slot is only electrically connected to x8 or x4 or worse x1 lanes and doing this is still following the PCIe specification. Yes a lot more people are using a GPU in a PCie x8 lane connection than a lot would presume because that is what port their PCIe is connected to has wired up.


Yes it also common for a x8 lanes to be wired from the chipset to a PCIe x16 slot but the chipset only have x4 lanes to the CPU lot of people incorrectly think the poor performance there is the PCIe x8 lanes from the slot to the chipset not the PCIe x4 lanes of the chipset to the cpu what is the true problem. The fact the chipsets in a lot of cases is PCIe x4 lanes when you use the chipset connect slot and people find a gpu there usable quite a bit of the time does also explain why thunderbolt 3/4 does kind of work ok. Same bandwidth bottleneck problem.

[muncrief]

Oh wow, it would be awesome if I could bind and unbind my AMD GPU to/from vfio and AMDGPU. I've always had to have my second GPU sit idle most of the time because I use it for my full passthrough Windows 10 VM, which I rarely use. The original plan was to primarily use the VM to run Band in the Box, but since BIAB can't be integrated with my Linux pro audio system I've found it to be mostly useless. So now I mainly boot up the Windows VM simply to make sure it doesn't get too out of date.

My dream is that one day wine will advance enough to run BIAB, but after so many years that dream is fading. However it can't solely be blamed on wine, as BIAB is a messy combination of very old code with new code tacked on. The whole thing needs to be rewritten from scratch but I doubt PG Music will ever do that. They also seem to be actively opposed to Linux for some reason, so I just stopped paying for updates in 2019 and moved on as best I could.

[tildearrow]

Amazing! Hopefully this means that if my card suddenly shuts off due to a slight input voltage instability it will come back up without reboot (even though my motherboard doesn't support hot-plugging but apparently rebooting manages to get the card back on).